increasing the comparison threshold without hurting the playback quality.

The playback quality is what you have to decide on with regard to the results. With regard to that quality, be careful with your algorithm, the threshold, and how you're implementing it. If you're using only 3 consecutive frames ( a, b, c ) to determine if b is "redundant," you may be "flattening" curves unnecessarily.

I.e., in the pix below (just an illustration of the principle,) if you approach the curve from the left, if your threshold is a bit too large, you may decide to eliminate point b, where point c may, in fact, be a better choice.

[attachment=27810:redundant_point.png]

Just some random suggestions - I understand your rendering algorithm is tied to the format of the keyframe data, but (as you likely know) it's driving up the size of the data structure. I know it's a decision among refactoring, code speed and data size. For your consideration (perhaps for future implementation):

- A matrix is 16 values. A rotation matrix (9 values), a translation vector (3 values) and a scale vector (3 values) are 15 values. A quaternion instead of a rotation matrix makes the structure only 10 values. Scaling is rarely needed, and eliminating that saves you 3 values ( perhaps 12 bytes/frame ).

- you're using long for your tick count. Do you ever have an animation that long?? In addition, it's a signed integer and you shouldn't be using negative tick counts. I.e., uint8_t or uint16_t may cover the max tickcount needed, doubles the value range and can save several bytes per keyframe. uint32_t, at a minimum, is no larger and doubles the range.

scaling, ... I don't think I will ever need it.

It's rare, but not unknown. It can be used for morphing, i.e., causing the mesh around a bone to "swell" or "shrink." Probably a good decision for now.

Another random thought - my brain works in the background sometimes, and it occurs to me that, because you're using a matrix at run-time for your animation calcs, (unless you decompose it) you may be linearly interpolating your rotations. If so, you should use the same algorithm in your preprocessing to determine redundant frames. I.e., if you SLERP to determine unneeded rotations in your exporter, you won't get the expected results if you do a linear interpolation at runtime.

N.B., SLERPing and LERPing over small angles won't provide greatly different results, but they are not identical. Just recommending you make decisions during preprocessing that reflect the results you'll end up with at run-time.

SLERP was necessary to use

Just a suggestion - if you're decomposing the matrix at runtime to slerp the rotations (and, I assume, interpolate the translation and scale), why not export the data as separate rotation, translation and scaling to begin with?