Let me explain. Opengl defines the X+ as to the right, Y+ as up, and Z- into the screen. I've gotten used to this coordinate system, and it's easy to think in. Except I have 1 issue with it, I've gotten myself stuck with 2 definitions of rotation matrices in my code, and I'm trying to unify them.

I don't have a problem with translation, so for now let's only consider the 3x3 rotation section of the matrix.

Suppose I want to draw an object, unrotated. The 3x3 rotation matrix is just identity:

1 0 0
0 1 0
0 0 1

I also represent the camera's current rotation as a matrix. I form the camera matrix out of the 'look', 'up', and 'right' vectors. Except, the unrotated camera looks into -z, so the unrotated camera matrix is:

1 0 0 //right (+X)
0 1 0 //up (+Y)
0 0 -1 //look (+Z)

I have a few dumb negative signs in my code so that when I'm applying a camera matrix, the above matrix acts like identity. But I want to unify everything, I want applying a matrix to just be applying a matrix. So I have a few options:

1. Switch to right handed: if +Z went into the screen, an unrotated camera, and an unrotated object would have the same identity matrix.

2. Don't form a camera matrix with up, left right (edit: brain fog/typo), and look vectors. Instead use up, left, and 'ass' vectors. Then the problem goes away, but I need to negate the backwards vector to get which way a camera is looking.

3. Something else?

How does everyone else solve or avoid this problem?

Thanks.

First of all, your camera matrix's up, left, and look vectors should be the columns, not the rows. That's probably one issue you might be having.

The main issue is that the direction of the matrix's transformation is different for objects and cameras. For objects, the matrix should transform points from object to world space. For cameras, the 'camera object matrix' transforms from camera space to world space, while what you want is the inverse camera matrix, the view matrix which transforms from world to camera space.

You will find that the camera's object matrix, not the view matrix will be the identity matrix when there is no rotation. If you respect the transformation directions and make sure that you are using right-handed coordinates to compute your viewing directions, you shouldn't have any inconsistencies.

In my engine, the camera's look vector is indeed the negation of the camera's object matrix's 3rd column.

Thanks for the reply.

First of all, your camera matrix's up, left, and look vectors should be the columns, not the rows. That's probably one issue you might be having.

Isn't it the opposite? Look at at gluLookAt: http://www.opengl.org/sdk/docs/man2/xhtml/gluLookAt.xml

It appears to:

f = normalize( center - eye ) (making F the 'look' vector)

Then does s = f x UP (making s the 'right' vector'

And creates the matrix:

s0 s1 s2 0 //right

u0 u1 u2 0 //up

-f0 -f1 -f2 0 // -look

0 0 0 1

Opengl orders in column major, so in RAM:

float matrix = { 
s0, u0 -f0, 0
s1, u1, -f1, 0
s2, u2, -f2, 0
0,0,0,1};
 

Is this correct?

The reason you're getting confused by that page is because the 4x4 matrix they are creating is the inverse of the camera's rotation matrix. Since for an orthogonal matrix its inverse is its transpose, they have their vectors in rows, indicating the inverse transformation.

Your code above looks correct to me.