Would be a lot more interesting to see the part where you set the view matrix, because I'd almost bet that you're using some lookat function and never considered that once you rotate more then 90° up or down just simply plugging (0,1,0) as "up" into it will be the completely wrong half space. The "real" up at 90° is either (0,0,1) or (0,0,-1) and beyond that it's (0,-y,z) or (0,-y,-z). You do NOT specify what's "up" in the world, but what's up for the viewer. They just happen to usually be in roughly the same direction and lookat happens to fix that with the cross products.
Good reason why just using tutorials and their code isn't enough. At some point you need to learn the background, why things are done that way, how they actually work and most of all, how they work together. In this case, understanding what that black box called "gluLookAt" (or the DX equivalent) actually does.
In fact, everytime there is a topic about rotation/camera problems, it's also a safe bet that there will be Euler angles. Sufficient for simple first person stuff, but somewhat useless for everything beyond that. Quickly checking Google for a camera tutorial, I know whom to blame. Unfortunately, everybody and his dog can put up tutorials on the internet and almost all of them use Euler angles.
Quick and dirty (OGL using a float[16], D3D using the provided matrix class), removed all other stuff (culling, translating, rotation around global axes, etc.):
//Ditch gluLookAt and just do it yourself with less math involvedvoid CameraGL::setView() { glMatrixMode(GL_MODELVIEW); float viewmatrix[16]={ Transform[0], Transform[4], Transform[8], 0, Transform[1], Transform[5], Transform[9], 0, Transform[2], Transform[6], Transform[10], 0, -(Transform[0]*Transform[12] + Transform[1]*Transform[13] + Transform[2]*Transform[14]), -(Transform[4]*Transform[12] + Transform[5]*Transform[13] + Transform[6]*Transform[14]), -(Transform[8]*Transform[12] + Transform[9]*Transform[13] + Transform[10]*Transform[14]), 1}; glLoadMatrixf(viewmatrix);}//For simplicities sake, we don't use a matrix lib and inefficiently abuse//OpenGL to do the matrix multiplication for usvoid CameraGL::rotate(float deg, float x, float y, float z) { glPushMatrix(); glLoadMatrixf(Transform); glRotatef(deg, x,y,z); glGetFloatv(GL_MODELVIEW_MATRIX, Transform); glPopMatrix();}//Same for D3D, except we don't manually inverse the matrix//(the generic inverse might be less efficient, but makes it more obvious//what's happening)void CameraD3D::setView() { D3DXMATRIX ViewMat; D3DXMatrixInverse(&ViewMat, NULL, &Transform); device->SetTransform(D3DTS_VIEW, &ViewMat);}void CameraD3D::rotate(float rad, float x, float y, float z, bool global) { D3DXMATRIX rot; D3DXVECTOR3 axis(x, y, z); D3DXMatrixRotationAxis(&rot, &axis, rad); D3DXMatrixMultiply(&Transform, &rot, &Transform); }
The whole point of the above is that moving/rotating objects is exactly the same as moving/rotating the camera, except that the matrix has to be inversed to be used as view matrix.