You should post the code that produces the red sphere too, so i can figure out what you try to do.

I assume you try to match physics to some animation - why so?

Also, if you control a physical object by changing its velocity, typically you do not care about acceleration. You care about acceleration only if you control the object by torque. E.g. for a question like "i want angular velocity of (10,0,0) - what torque do i need to achieve this?" Here acceleration matters, but by setting velocity directly you bypass any current physical state and just change it.

(I assume that's one cause of what you mean with oscillation)

At this line:

vec3 angularVelocityVector = GEMultiplyVectorAndScalar(GEAxisFromQuaternion(quaternion), GEAngleFromQuaternion(quaternion));

You calculate velocity from orientation after rotating 3 times. This makes no sense. You should calculate it from a rotation.

Note the difference between both terms (comparing angular to linear stuff here):

Orientation is like position or a point, describing state in space.

Rotation is like a displacement or vector, describing a difference between two orientations, or how orientation changes if we rotate it.

Differentiating between those two terms also when thinking about a problem helps a lot to deal with 3D rotations   I do not differentiate between 'point' and 'vector' - that's always easy, but for 'orientation' vs. 'rotation' it really helps.

So i assume you want to calculate a single rotation that has the same effect than applying your 3 hardcoded rotations in order. And then you want to convert this single rotation to angular velocity so a physical body matches the result. Is that correct? This is what my 'some code' snippet does.

angleEulerRadians = GEAddVectors(angleEulerRadians, GEMultiplyVectorAndScalar(angularVelocityVector, globalTimeStep));

This makes no sense. If you need to convert to euler angles you need a function that does the conversation (worth its own topic), but i assume you don't need them so avoid them to do yourself a favour.

You should do this instead:

Give your final orientation to glLoatMatrix,

or give a rotation to glMultMatrix or glRotate (the latter using axis and angle, not eulers)

(assuming you work in global space and you are not inside a transform hierarchy or something)

3 hours ago, Jonathan2006 said:

I just thought of something. Even if I use something like quatCur - quatOld should the angular oscillating stop if I normalize the quaternion like you did in your code?

No, actually this normalization has no effect and should be removed. 

3 hours ago, Jonathan2006 said:

I am still trying to wrap my head around quaternions.

You and everybody else  Thinking of it as an axis and angle rotation is good enough to work with them. It's more important to understand what you can do with various rotation representation, e.g.:

Quat: Easy to interpolate, easy to convert to axis and angle

Rotation Vector (axis times angle): You can sum multiple rotations, no angle limit

Matrix: You can also represent scale and skew

Euler angles: Good for numerical user interaction

The red wire sphere code is the “mat4 rotateMatrix” that I posted further above. Thank you JoeJ for your time. I just don’t know enough to figure this out. Below are some things I have learned.

After skimming through some game developer physics books, euclideanspace.com, and your posts here’s what I think I have learned so far...
1) Axis Angle can be used to represent the length and direction of angular velocity/rotations in quaternions and matrices.
2) Multiplying two quaternions is just like multiplying two matrices.
3) Angular velocity can be represented in a 3x3 skew matrix.

What went wrong
1) I could never get any of my axis angle functions to work with more than one integrator.
2) Not sure about multiplying quaternions since I could never get the correct output after multiplying.
3) I tried multiplying several skew matrices together and the final matrix had all zero values.

I used the GLM library for most of my quaternion and matrix code but I was never really able to debug the axis angle quaternions. I tried axis angle matrices from euclideanspace.com and I did find a bug in the GLM code that I fixed in my code. The output of angular velocity in the matrices never came out correctly after I added more than one integrator.

Since nothing worked I will have to use my old unreliable code below:

vec3 angleRadians = GEEulerAngleFromMat4(matrix);
vec3 angularVelocity = (angleRadians - oldAngleRadians) / globalTimeStep;
vec3 oldAngleRadians = angleRadians;

Hope this helps anyone else with this problem.

Thanks again,
Jonathan

10 hours ago, Jonathan2006 said:

1) Axis Angle can be used to represent the length and direction of angular velocity/rotations in quaternions and matrices.

Axis and Angle: Axis has always unit length.

"Rotation Vector" = axis * angle (so just 3 numbers instead 4) Unfortunately i don't know a common name for this - "Rotation Vector" is just my own term.

Both of them can represent angular velocity because both support multiple revolutions, Rotation Vector is preferred because you can sum up multiple velocities before you perform any rotation.

10 hours ago, Jonathan2006 said:

2) Multiplying two quaternions is just like multiplying two matrices.

Yes (as long as they both represent a rotation / orientation. Exception is e.g. a scaled or sewed matrix. Quats can't handle this.)

10 hours ago, Jonathan2006 said:

3) Angular velocity can be represented in a 3x3 skew matrix.

Don't know.

10 hours ago, Jonathan2006 said:

1) I could never get any of my axis angle functions to work with more than one integrator.

You typically use only one integrator: You sum up all forces that affect an object, calculate velocity and integrate it for the time duration you want. In your case it's easier as you don;t deal with forces i guess. 

I see you integrated 3 times in your code but that's wrong - you want one single rotation that does the same as your 3 hardcoded rotations. So one single angular velocity calculated from 3 ordered rotations to integrate.

I would do this for example for a problem like this: I have animation data with euler angles (which are 3 ordered rotations each as well), and i need to feed the physics engine of a game so a simulated rigid body matches the animation. I assume your problem is equivalent, and all my response is based on that assumption. (Why else would you want to calculate angular velocity? You don't tell me....)

10 hours ago, Jonathan2006 said:

2) Not sure about multiplying quaternions since I could never get the correct output after multiplying.

If you have some rotation matrices, multiply them in order, the result must be the same if you first convert each matrix to quat and multiply them in the same order. Check this out to be sure your math lib works as expected.

10 hours ago, Jonathan2006 said:

3) I tried multiplying several skew matrices together and the final matrix had all zero values.

You'd need to look at the numbers, probably there are zero rows or columns that cause this.

10 hours ago, Jonathan2006 said:

Since nothing worked I will have to use my old unreliable code below:

vec3 angleRadians = GEEulerAngleFromMat4(matrix);
vec3 angularVelocity = (angleRadians - oldAngleRadians) / globalTimeStep;
vec3 oldAngleRadians = angleRadians;

Hope this helps anyone else with this problem.

Noooo!

Angular velocity is not the difference between Euler Angles. This may work in special cases, e.g. two angles are zero or cancel each other out due to gimbal lock. But in general it's terribly wrong. The reason is Euler angles has a specified order. The angle numbers alone don't know that order, but you can't interpret them without defining that order. Taking the difference ignores all this.  (I remember a game physics book that did that same mistake but not the title. If you have it, burn it.)

Rule of the thumb: Avoid using euler angles unless you have a very good reason.

Understanding 3D rotations requires practice and it's hard to communicate. To get better help you need to describe very precisely what your problem is in practice, and why you need to do conversations from a reference to something else.