Using an ellipse projected on a sphere to define the allowed rotations. With also some approximate conversion from euler angles into this representation.

Decompose the quaternion into swing and twist representation. Constraint the swing to this area. And smoothly blend towards the desired rotation of the lookat.

Sorry if this is a bit vague :) You could even use plain euler angles, but they are evil and give problems. Also they don't look so smooth. The projected ellipse gives very pleasing and smooth results for things such as the human head, eyeballs and upper spine bones etc.