I have a skinned + animated model. It's a character model. I'm trying to get his head to look at an object in world space. This is the problem. If I take his head local matrix and build a matrix that looks at the world space object, that doesn't work. This is because when the matrix gets combined with the parent the matrix isn't looking in the right direction.
My final matrix looks like this..
Final = InvBindPose * LocalBone * CombinedParents;
So, how do I get my localBone matrix to be the correct one to look at an object in world space? Do I need to invert the combinedParents matrix and then re-translate it from the combinedParents matrix? to get it in the right position after the look at is applied?
From your equation, I assume you are using the v*M notation, and will use the same below.
First, compute a standard LookAt matrix that orients the front vector in the local bone space towards the desired world space direction. For reference, see e.g. MathGeoLib float3x4::LookAt, and its derivation.
Let's denote the lookat matrix by L. We want this matrix L to be the local->world matrix for the bone, but since the bone has parent nodes, we have a sequence local->parent->world. The problem is to compute such a local->parent matrix M for the bone, so that the local->world matrix for that bone will end up to be L. You say you know the parent->world transform (your CombinedParents matrix), so let's denote that by P. Then, we have the equation
local->world == local->parent * parent->world, or in terms of matrices above L == M * P, where we solve for the desired M, M = L * P^(-1).
Therefore, compute the local->world LookAt matrix L, then compute the world transform of the parent, P, inverse it, and compute L * P^(-1), and set that as the local->parent matrix M for the bone.
The head is correctly facing forward always... however as I rotate the character in the world, his head is always forward facing on the body. I am trying to get him to look down world Z instead of model/mesh space Z. So, I tried multiplying LookAt * InvWorld, but that didn't work. What should I do with my WorldMatrix to multiiply it with my new LookAt to get it in world space always?
Your bone matrix transforms from the bone's local space to its parent space. If you combine the transforms W*B1*B2*B3*...*BN, you will get a matrix which transforms from the local space of bone BN to the world space. W is a matrix that transforms from the local space of the object to the world space. B1 is the root bone's local->parent matrix. (Depending on your setup, you may or may not have a separate matrix W, and might just treat B1 as W).
To transform from world space to the bone's local space, compute the matrix M^-1, where M = W*B1*B2*B3*...*BN. That matrix will take world space points/direction vectors into the local space of bone BN.
If that's still not working, try pasting some code if someone can spot something that's off.
That worked perfect. Just knowing what I am doing is correct helped a lot as I could focus on other stuff that may have been wrong. I think the biggest thing is my model is coming from max which has a different coordinate system. So, I'm basically trying to figure out how to change my look at matrix to match it's coordinate system in the bind pose so the face is facing the right direction. I think I can take it from here. I'll PM you if I have any other questions.