So I have a nice animation blending system that allows me to transition from one animation to another seamlessly.

I was originally blending the position, rotation, scale individually but then I rewrote it to blend the matrices themselves, both as a slight optimization and as a fix for a horrible bug I won't go into.

Here's the code I use to blend two matrices from transform 0 to transform 1.

transform[0] = (transform[0] * (1.0f - m_transitionWeight)) + (transform[1] * m_transitionWeight); 

This is very similar to the math inside the GLSL skinning shader so I'd think this would work.

//skinning
mat4 transformedMat = bones[int(boneIndices[0])] * weights[0];
transformedMat += bones[int(boneIndices[1])] * weights[1];
transformedMat += bones[int(boneIndices[2])] * weights[2];
transformedMat += bones[int(boneIndices[3])] * weights[3]; 

Now my animation went from blending perfectly, to having my character's legs shrink a bit when it's part way in the animation. It's the Doom 3 space marine aiming up, then aiming straight. This blending system makes his body turn to positions in between aiming up and straight to let him aim in any direction, but with my matrix math tweak his legs randomly shrink a bit as well.

I did some reading and I guess this has something to do with the rotations needing to be properly slerped, which was happening when I was interpolating rotation, position, scale individually. But when I tried to blend matrices, the rotation slerping was gone. I'm still a bit fuzzy as to why this works properly for vertex skinning.

So I have a nice animation blending system that allows me to transition from one animation to another seamlessly.

I was originally blending the position, rotation, scale individually but then I rewrote it to blend the matrices themselves, both as a slight optimization and as a fix for a horrible bug I won't go into.

how can interpolating matrices possibly be quicker than interpolating quaternions (as i presume you did in your first version)?

also lerping matrices for animation cannot work: consider these two 2x2 matrices, both are in the orthogonal group:

0 1 1 0

-1 0 and 0 1

which represent a clockwise 90 degree turn and the unity matrix. simply lerping them would yield at transition weight tw = 0.5:

0.5 0.5

-0.5 0.5

which has a determinant of 0.5, which, among other things, means that lenghts aren't preserved. which probably causes your problem described.

what you should be doing is just sticking with quarternions; instead of the costly slerp you can also use the so called nlerp (n stands for normalized as compared to spherical), provided the interpolated angle isn't too big.

heres the nlerp im using in my vertex skinner:

Quarternion Mesh::Nlerp(Quarternion qa, Quarternion qb, float t2) {
	Quarternion qm;
	float t1 = 1.0f - t2;
	qm = qa*t1 + qb*t2;
	float len = sqrtf(qm.x*qm.x + qm.y*qm.y + qm.z*qm.z + qm.w*qm.w);
    if (len<=0) LOGMSG("faulty quaternion");
    qm /= len;
	return qm;
}

as you can see, its very easy and cheap to nlerp. converting q's into matrices is a bit more tedious, but this procedure is the only way i know of (which isn't saying there aren't any other more clever schemes out there), and widely used to my knowledge

EDIT: ah btw, by orthogonal group i mean they have determinant 1, or simply put are regular rotation matrices (sorry for the math blabber).

As Tasche already explained, is a pure rotation matrix required to be ortho-normal. It is possible to interpolate rotations without using quaternions, but following Dave's argumentation (in this PDF located on its famous Geometric Tools site), performancewise it is a slightly less effective way.

Well, looking a the interpolation methods slerp and nlerp, both have their advantages as well as disadvantages. nlerp is faster to compute (always good) and is commutative (less problematic when arranging animation tracks). But it lacks linearity in the interpolated angle, or, in other words, constant angular velocity. In the extreme case the resulting quaternion will vanish. That are the reasons why nlerp should be used with relatively small difference angles only.

So I have a nice animation blending system that allows me to transition from one animation to another seamlessly.

I was originally blending the position, rotation, scale individually but then I rewrote it to blend the matrices themselves, both as a slight optimization and as a fix for a horrible bug I won't go into.

how can interpolating matrices possibly be quicker than interpolating quaternions (as i presume you did in your first version)?

Well it was faster since I didn't have to convert from matrix form, to the three components separated, interpolate those, then back to matrix form.

With the new way I was just interpolating in Matrix form.

I'm actually doing a huge rewrite now where all the bone data and transforms are stored as the three components separated and then right before rendering in the shader I'm converting the final pose to 4x4 matrices and sending that down.

Well it was faster since I didn't have to convert from matrix form,to the three components separated, interpolate those, then back tomatrix form.

With the new way I was just interpolating in Matrix form.

I'm actually doing a huge rewrite now where all the bone data and transforms are stored as the three components separated and then right before rendering in the shader I'm converting the final pose to 4x4 matrices and sending that down.

thats the way i do it. the animation data is already loaded as quarternions, then gets interpolated that way, and only before final composition with bonematrix do i convert. (also cuts load time a wee tiny bit i guess)

Given the Tasche example of quaternion Nlerp, he is normalizing in that function.

If i nlerp two keyfrmes from one and two from other animation/track, can i pass with just one normalize?

quat walkQuat = nlerp( walk.key.quat, walk.key[i+1].quat, tm ); // nlerp without normalize
quat sprintQuat =nlerp( sprint.key.quat, sprint.key[i+1].quat, tm ); // nlerp without normalize
quat finalQuat = nlerp( walkQuat, sprintQuat, tm ).normalize();

Thank you for your time.

^ Bump.

I haven't tried that. It could supposedly work. My math knowledge here is a bit shaky, I just know how to use glm and how to transform things by other things. I've had many many hours spent trying to fix bugs only to later realize it's because my math was wrong

I guess I'm slowly learning doing something fun rather than taking a Linear Algebra class and having to do annoying homework.

Given the Tasche example of quaternion Nlerp, he is normalizing in that function.

If i nlerp two keyfrmes from one and two from other animation/track, can i pass with just one normalize?

quat walkQuat = nlerp( walk.key.quat, walk.key[i+1].quat, tm ); // nlerp without normalize
quat sprintQuat =nlerp( sprint.key.quat, sprint.key[i+1].quat, tm ); // nlerp without normalize
quat finalQuat = nlerp( walkQuat, sprintQuat, tm ).normalize();

Thank you for your time.

short answer: no.
post back here if you need the long answer.

post back here if you need the long answer.

Yes, i might learn something.