I'm a bit more ashamed than you not knowing how a physics solver actually works :)
Newton engine builds the jacobians, what i give as input for a motor is angular acceleration about an axis.
In theory we would need only need one axis and angle to get any rotation we want, but newton likes to do this for 3 orthonormal axis, so everything is defined and stable. I guess something similar applies to you.
I refer to the orientation build from this 3 axis as 'constraint space' (not sure it's the proper term), and the primary axis where the moter acts as 'error axis' (axis from Dirks 'qr' quat).
The problem is: Newton wants constraints order independent, which makes sense because we want them to be solved at the same time.
But it is impossible to decompose a rotation to multiple rotations so you can combine (multiply) them together in any order and get the initial result. Whatever order you take, you get a different result.
So my trick here limits the problem, because at least the measurement of angles is guaranteed to be order independent, and by aligning constraint space with error axis any error becomes negligible.
However, if your (or Dirks) solver solves the constraints in given order, then you don't have this problem.
The problem in your video looks like a jump over 180 degrees or similar gimbal lock issue coming from eulers. I guess not your jacobians are wrong but more likely the angle measurement.
But i also can see some unnatural twisting, also a typical euler issue. Euler angles == evil. Both Dirks or mine suggestion should fix those things.
Other than that it looks good - seems you're not far off.
I tried to work on twist / swing again, but Newton seems somehow broken at the moment due to heavy changes for new softbody feature. I need to clarify this...