Try running qu3e and record an input matrix and its corresponding output. Then, setup a test case where you use the exact same floating point values, and try to transform your inertia tensor. You should verify where the values are the exact same, or transposed, or otherwise incorrect.

After weeks of having not the slightest desire to continue fiddling around with this, I gave it another go today and finally managed to make it work.

I settled for column vectors as convention and adapted my matrix calculations accordingly. The error was somewhere else though. I integrated the angular velocity using

Orientation += 0.5 * Orientation * AngularVelocity

instead of

Orientation += AngularVelocity * Orientation * 0.5

I definitely learned my lesson to pay more attention to the order of operations.

Thanks for all your help!

This happens to me as well sometime. I think the reason is that you tend to look for mistakes in the math rather than for simple programming/typo mistakes. :)

Haha that's a pretty good one! Thanks for sharing. Very entertaining. We have all struggled with this exact kind of mistake. Back when I intern at thatgamecompany I had a week long problem between row vs column notation. Nobody in the company understood which was which, and instead just had "a consistent ordering" in the codebase. But it was a problem when attempting to port in code with unknown ordering... Or implement papers where they have different notation compared to the code.

For scalar multiplication one thing I found that helps is to only have on operator defined, so 0.5 * vec can compile, but vec * 0.5 can not compile. This kind of thing has prevented order swappings for me before, especially when porting code between code bases.

---

One time I had a typo in the collision constraint code back in school. The typo was accidentally re-using an old vector in the solver, as opposed to the correct one, because of a typo. Funny thing was numerically each vector was nearly the same, so long as the constraint geometry was close to the origin.

But, once shapes were far away from the origin the error would get larger. The result was whenever the player bumped into tables (the only dynamic objects) and woke them up, the collision constraints between the tables and the floor would make the table jiggle in a little spiral pattern. As if it were walking in a tiny circle. Really far away from the origin would make tables swirl violently.

It took a month to find and fix that bug! I audited basically my entire rigid body engine. For the longest time I thought it was a bad inertia tensor. Turned out to just be a typo in the solver.

Glad someone at least got a chuckle out of it. I've had my fair share of silly programming errors over the years, but this was one of the tougher ones for me to spot, that I remember, and definitely one of the most frustrating.

Anyway, it's working great now: