...NOT, it was just three classics:
1.) vec2 * mat2 is not the same as mat2 * vec2
Rotate left against rotate right. -> contacts was wrong rotated.
2.) Cross(1.0f, normal) is not the same as Cross(normal, 1.0f).
Same as above, inverted rotation...
3.) There was always a friction impulse going on, even with friction set to zero.
A classic wrong arguments orders bug: Clamp(min, value, max) is not the same as Clamp(value, min, max) and i had:
contact->tangentImpulse = Clamp(oldTangentImpulse + tangentImpulse, -maxFriction, maxFriction);
Lessons learned:
- Porting from own old sources should never be underestimated, especially when there was no operator overload used or supported!
- Check your math library before doing any useful with that, especially matrices and rotations. Make Units tests or something...