I'm really sorry, but I, nor anybody else in my class (or even the next class) has learned 3D Cartesian geometry/analytic geometry.
Everything we have been doing so far is only 2D (with the exception of formulas for cubes and pyramids).
But if you don't know anything about 3D geometry how do you expect to understand 3D matrix transformations/projections, sense of ordinary space, all that stuff? Not that I'm saying it's impossible but it will be difficult without a knowledge of the basics. Otherwise magic constants (such as the 20's in your code) start appearing and you have no idea why.
Anyway. The D3DXMatrixLookAtLH function takes as an input a 3D coordinate which specifies the location (in "world" space) of the camera, and a 3D coordinate which specifies the location of the target (thus the direction in which the camera is looking can be obtained by a subtraction). The third parameter is the up vector but it can be kept to (0, 1, 0) until you want to perform "roll" rotation (i.e. rotation along the Z axis).
In your code the camera's target is derived from the mouse position in an incorrect manner. What happens is that your camera's target (what the camera is looking at) is rotating along a non-obvious curve centered on the origin (because you messed up with the angle calculation). I suspect you want the camera target to rotate in a graceful curve as you move the mouse around (like a spherical magnet would slide all around another).
You need to use spherical coordinates for this, (this is why you need to know 3D geometry) which map two angles and a radius onto a sphere's surface. The formulas are not obvious to the beginner but work (more on wikipedia):
For a horizontal angle Phi and a vertical angle Theta (both in radians), and a radius R, the corresponding point on the sphere (centered on the origin, translate if needed):
X = R * cos(Phi) * sin(Theta)
Y = R * Sin(Theta)
Z = R * sin(Phi) * Sin(Theta)
(assuming the Y axis goes upwards).
Now if you put this in your code, instead of the "ant around a circle" result Apoch described earlier, you will get an "ant around a sphere" result. Now if you add MoveX to LookX and MoveY to LookY and MoveZ to LookZ, then you will have translated the rotation center onto the camera's position, which means you will be able to look around in every direction by moving the mouse around (like in FPS shooters, for instance). Finally, the radius of the rotation does not matter because it will not change the direction the camera is facing (in the LookAt function, the direction ends up being normalized, so any scaling factor is cancelled out).
As for the the magic constant in the code which "sorta kinda worked", I suspect it simply happened to reduce the visible error to a minimum (while still being completely wrong).
So the final correct code would be:
LookX = cos(angleX) * sin(angleY)
LookY = Sin(angleY)
LookZ = sin(angleX) * Sin(angleY)
MoveX, MoveY, MoveZ as usual (keyboard is used to move around)
And:
D3DXMatrixLookAtLH(&matView,
&D3DXVECTOR3 (MoveX, MoveY, MoveZ), // the camera position
&D3DXVECTOR3 (LookX + MoveX, LookY + MoveY, LookZ + MoveZ), // the look-at position
&D3DXVECTOR3 (0.0f, 1.0f, 0.0f)); // the up direction
And there you have it, you reinvented camera rotation in its simplest form. With this code you should be able to use the keyboard to move around the world, and use the mouse to naturally look around (which is what you wanted, right?)
Also, make sure you put bounds on the Theta (angleY) angle so that the camera cannot "flip vertically" because then everything will be inverted. In real life this would correspond to looking forward, then looking upwards, until you bend your neck backwards (and same for looking downwards), which results in an extremely unintuitive sense of direction, so for player sanity we usually ensure -pi/2 < angleY < pi/2, although this may depend on the API (conventions and stuff). But that's just a minor detail.
-------------------
I may have made mistakes in the post above, but I think this is what you were looking for. If not, then I did not understand your problem so if it is so can you reformulate it in the simplest way possible (what do you want to achieve?)
0
