The problem is that once the camera is no longer pointing down the z-axis of the identity matrix, rotations upon world space no longer look like "camera" rotations. Example : On my keyboard the z and x keys are supposed to control the camera''s roll. Which they do if the camera is at identity (because I rotate around the z-axis). I rotate the camera around the Y-axis by 90 degrees - now any change along the z-axis affects not the roll but the pitch of my camera. This problem is obviously as old as the hills, but how to solve it ? Quaternions, while probably the best solution, are not what I''m after, nor gluLookAt type solutions, just raw matrix stuff.