So an update!

Things are more stable when the iterative solver just stops, when the closing velocity along the contactnormal gets below a certain threshold. I've had to disable friction, since it results in a subtle slide, that I haven't debugged yet. I've also checked the first transformation of the inertia tensor, and at least for the first frame everything is correct. I have to check whether there is some math error once things start to rotate, but that's on the todo. In terms of code i just check if the dot product of the closing velocity and the contact normal is above -0.01 and continue if it is.

The box seems to perfectly balance on that edge, which is a bit unrealistic. Will have to change the orientation to like 30 degress and see what happens.

Sooooooo….. I've redone a lot of stuff….. It seems like there was problems with the quaternion math.. Both in retards the the transformation matrix and the actual addition of the rotational velocity to the orientation during the integration step.

This is the closest i've ever been (it still doesnt work as it should in other testcases):

So far in every test case, I've seem to have fixed it. I'm catious though, I might end up regretting celebrating too early.

A lot of problems were related to the quaternion representing the orientation. In part there were bugs in the way the transform matrix got computed from it. And in part the way the angular velocity was used to update the orientation was also problematic. It was related to the order and method of multiplication of two quaternions.

That is also why both the Ian and the Cookbook method didn't resolve it. My own version of Ians method works really great (again I hope I don't jinx it!).

I'm so happy.

It only took me a month to find these stupid bugs. Now i have a somewhat working physics system!!!

Thanks to any readers and posters. I'm gonna leave this comment for future developers. Always check your math methods with online calculators, and check them twice if you need to ;-)

BadProgrammingGuide said:
And in part the way the angular velocity was used to update the orientation was also problematic. It was related to the order and method of multiplication of two quaternions.

It might be still problematic even after fixing the bugs.

A common problem is small cross product magnitudes. Too small to normalize, and resulting angular velocities or accelerations become of low accuracy.
But this can be addressed with small angle approximations, and with quaternions this works really well: https://theorangeduck.com/page/exponential-map-angle-axis-angular-velocity

@JoeJ So far the fixes are holding up. There are only some minor stuff in terms of stability (like some objects very slowly crawling), which I think is something I'd correct in the actual engine and is typical for a lot of engines in general. But I'll definitely look into it. Maybe I need some checks here and there in the algorithm :-)