I definitely need to brush up on this stuff to have it clear in my head. Thanks haegarr, hopefully we've helped the OP

@Yezu666,

make sure you follow what haegarr said about normalizing the vectors.

Although I haven't noticed so far, I think there are errors in my LookAt function. AFAIK the resulting matrix should be orthonormal and the scaling effect after transposition suggests it is not. I must look into it. However I would be very happy if You could help me. Here is the code:

Please bear in mind that some aspects of the implementation are not clear yet. So I must guess some things...

public static TransformationMatrix calculateLookAtMatrix(Vector3d eye, Vector3d center, Vector3d up)
{
float [] matrix = new float[16];

Vector3d forward = Vector3d.sub(center , eye);
forward.normalize();

I assume that Vector3d.sub(a, b) computes a-b (in this order).

Vector3d side = Vector3d.cross(forward , up);
side.normalize();

Vector3d plane_up = Vector3d.cross(side , forward); plane_up.normalize();

Normalizing "plane_up" is not strictly necessary, because both arguments "side" and "forward" have unit length and "side" is guaranteed to be orthogonal to "forward".

matrix[0] = side.getX();
matrix[1] = plane_up.getX();
matrix[2] = -forward.getX();
matrix[3] = 0.0f;

matrix[4] = side.getY();
matrix[5] = plane_up.getY();
matrix[6] = -forward.getY();
matrix[7] = 0.0f;

matrix[8] = side.getZ();
matrix[9] = plane_up.getZ();
matrix[10] = -forward.getZ();
matrix[11] = 0.0f;

matrix[12] = 0.0f;
matrix[13] = 0.0f;
matrix[14] = 0.0f;
matrix[15] = 1.0f;

Now things become a bit complicated.

1. You're negating (solely) the forward direction, so your camera is defined to look along its local negative z direction, right?

2. Because you're about to compute the transposed matrix, from the indexing I conclude that you're using the column major layout, right? I mean that the matrix indices are arranged like so:
[ 0 4 8 12 ]
[ 1 5 9 13 ]
[ 2 6 10 14]
[ 3 7 11 15]

If that is right it would further mean that you are using column vectors.


Vector3d neweye = new Vector3d(eye);
neweye.reverse();

TransformationMatrix ret = new TransformationMatrix(matrix);
ret.multiply(TransformationMatrix.calculateTranslationMatrix(neweye));

The ret.multiply computes probably R[sup]t[/sup] * T( -eye ) here. This order would be correct if and only if you're using column vectors. Do you do so? Unfortunately, nothing in the shown code gives a clear criterion whether you do so.


return ret;
}

Nothing to say here

It looks like it's already been addressed, but:

2. The handedness affects the order in which you do the cross product

Actually, (spatial) handedness has no effect on how the cross product is computed or how the arguments should be ordered.

3. By row-major and column-major I meant the same thing as row-vectors and column-vectors (I've always heard of the term row-major and column-major referred to matrices)[/quote]
Although these terms are commonly conflated, they refer to two completely different and unrelated things (as haegarr explained above).

Thanks jyk and haegarr for clearing this up

EDIT: can you go into a little more detail please?

From Wolfram Mathworld... http://mathworld.wolfram.com/CrossProduct.html

It says that the way the cross-product is computed there assumes the right-hand rule. Now I'm confused.

EDIT: can you go into a little more detail please?

From Wolfram Mathworld... http://mathworld.wol...ossProduct.html

It says that the way the cross-product is computed there assumes the right-hand rule. Now I'm confused.

I'm not sure why that specification is made (perhaps it's because the example in the diagram is with respect to a right-handed system).

In any case, the cross product is computed in the same way regardless of handedness; the 'right-hand rule' and 'left-hand rule' are just mnemonics to remember in which direction the cross product will point in a coordinate system with a given handedness.