LOL. Euler angles come with the problem of gimbal lock. Proper use of quaternions fixes that problem

So does proper use of matrices. Quaternions don't fix that because they are quaternions, they fix it, because they aren't Euler angles. The subset of quaternions used represent one rotation around one axis. Hooray, so does a rotation matrix. They are completely interchangeable. Show me any implementation using quaternions to "fix gimbal lock" and I can show you the very same implementation using matrices that does exactly the same (without having to constantly convert stuff back and forth between 3(!) representations of the same thing).

Yes, quaternions are neat if you want to save memory or have to concatenate a lot of rotations (for example skeletal animation). They are not "the only hope against gimbal lock", yet unfortunately way too many people hear "gimbal lock" and have the knee jerk reaction of "i must use quaternions, quaternions are fairy dust". My personal reaction is more like "must store orientation in useful way (rotation matrix), Euler angles suck" (except maybe for fps-style cameras, which I'd consider the limit of their usefulness).

Use them where it makes sense and because you understand WHY you are using them, not because people on the net say it's a magical silver bullet against gimbal lock (which I often take as a sign of "copy/pasted it from tutorial without understanding and now tells everybody to do the same"). I seriously fail to see how they are worth the conversions and overhead for something like a camera class.

In Method 2 you're not using the 'angles' parameter. Try this:

void Camera::Rotate(glm::vec3 angles){
glm::quat quat_x = glm::gtx::quaternion::angleAxis(angles.x * (180.0f / float(PI)), glm::vec3(view_matrix[[0][0], view_matrix[1][0], view_matrix[2][0]));
glm::quat quat_y = glm::gtx::quaternion::angleAxis(angles.y * (180.0f / float(PI)), glm::vec3(view_matrix[[0][1], view_matrix[1][1], view_matrix[2][1]));
orientation = quat_x * quat_y * orientation;
glm::gtc::quaternion::normalize(quaternion);
}

Note that you don't need to get the Euler angles from the original orientation quaternion - you're just building 2 new quaternions and applying them to the original rotation.

This still rotates stuff but it still rolls as well; which makes sense because I only fill angles x and y components from the mouse for rotation.

The most likely reason it rolls is that for building a quat from Eulers glm is probably applying rotations in the order x,y,z. What you want for a fps style camera is y.x.z (and that's what you do in your second method). The reason I point out that Eulers suck is that they do NOT represent one unique orientation, but _six_ possible orientations depending on the completely arbitrary choice of the order in which you apply them.

Even using quats/matrices you have that problem every frame when you turn mouse movement into rotations (which to apply first?). You can use the mouse coordinates to create a single rotation axis instead (normalized_delta_x * axis_up + normalized_delta_y * axis_right), but that would be counter productive for your needs.

edit: this needs some corrections and explanations...

Also at this point you could just store the cameras transformation matrix and replace your entire Rotate-function with

[s]orientation = glm::gtx::euler_angles:_yawPitchRoll( angles.y, angles.x, 0 ) * orientation;[/s]

[ Actually you can't, because you'd run into the same issues. If you look up 45° in the previous frame and then turn 180° right like this, you end up looking backwards and 45° down (if the function does it the way I expect it to) ]

[s]or[/s] just manually apply the two rotation matrices (angles.y around (0,1,0) and angles.x around (1,0,0) ) as

[s]orientation = rotationX * rotationY * orientation[/s]

[ again fell for it myself... for fps style camera, the whole point is to always rotate around "global" up first (and before your existing transformations) and around "local" right second (and after all previous transformations).. and maybe local "forward" if you want some leaning effect, so the correct order for this special case is:

orientation = rotationX * orientation * rotationY

One thing to always have in mind: matrix multiplication is applying transformations right to left and each one obviously changes your local coordinate system (ie. matrix). By doing rotationY first, you apply it while "local up" is still "world up" (ie. identity matrix).

This should kind of make it obvious that the order of rotations matters and why it simply won't work with just storing/summing up three single angles except for the trivial special case of typical fps style cameras, as long as you apply your angles in the order y,x,z. ]

Bottom line: there is no good reason to deal with either Euler Angles or quaternions for this kind thing (unless fps style is all you want to support).

for the non-fps cameras I stuck with what I had which was simply:


orientation = glm::quat(angles) * orientation;
// then normalize


that continues to work and thanks toTrienco's wonderful explanation, I stopped using my view matrix vectors as axes for rotation and also read up on the order of operations and how it affects quaternions and voila. I now have a non-fps and a fps camera.

Thank you to everyone so much for so much help! Not only did I get a much improved camera class, but now I have a much better understanding of the usefulness of quaternions as well. (and when they're not useful! )