I switched everything to quats.. and all is well except one thing.. ill try to explain the best i can
I build a rotation matrix from a quaternion and use it as such for all my game objects

screenSpace = projMat * camMat * translateMat * rotateMat * scaleMat * positionVec

and for my camera

camMat = rotateMat * translateMat * camOrigin

where rotateMat is given by the following

NSMatrix4Df NSQuaternion::getRotationMatrix() const
{
	NSMatrix4Df retMat;
	retMat.setRow(NSVec4Df(getRightVec(), 0.0f), 0);
	retMat.setRow(NSVec4Df(getUpVec(), 0.0f), 1);
	retMat.setRow(NSVec4Df(getTargetVec(), 0.0f), 2);
	return retMat;
}

NSVec3Df NSQuaternion::getRightVec() const
{
	return NSVec3Df(1.0f - 2.0f*y*y - 2.0f*z*z, 2.0f*x*y - 2.0f*w*z, 2.0f*x*z + 2.0f*w*y);
}

NSVec3Df NSQuaternion::getUpVec() const
{
	return NSVec3Df(2.0f*x*y + 2.0f*w*z, 1.0f - 2.0f*x*x - 2.0f*z*z, 2.0f*y*z - 2.0f*w*x);
}

NSVec3Df NSQuaternion::getTargetVec() const
{
	return NSVec3Df(2.0f*x*z - 2.0f*w*y, 2.0f*y*z + 2.0f*w*x, 1.0f - 2.0f*x*x - 2.0f*y*y);
}

I then have a rotate function that when I call rotates the object about a given axis - shown below

void NSQuaternion::rotate(const NSVec3Df & pAxis, float pAngle)
{
	NSQuaternion localRotation, temp(pAxis, pAngle);

	float halfAngle = sinf(DegreesToRadians((pAngle/2.0f)));
	localRotation.x = pAxis.x * halfAngle;
	localRotation.y = pAxis.y * halfAngle;
	localRotation.z = pAxis.z * halfAngle;
	localRotation.w = cosf(DegreesToRadians(pAngle / 2.0f));

	(*this) = (*this) * localRotation;
	normalize();
}

This works - but I'm confused because when I use this rotation function it rotates the object about its Local axis... so if I give it the vector (0,0,1) and the angle 45 it will rotate the object about its local z axis by 45 degrees no matter which way the object is oriented... if I change the last line (before normalize) to

(*this) =  localRotation * (*this);

it then rotates objects correctly about the worlds z axis if i give it (0,0,1) rather than its local axis. The problem is that with this version the targetVector that is returned by the function getTargetVec is not actually the direction the object is facing... hence...

when I use the second version the camera is all messed up..

Anyone know how to use a quaternion to store rotation information so that the same rotate function can be used by a camera as by an object? I would like both the camera and the objects to be able to rotate around world space vectors rather than their local space

If I want a rotation around a local space axis I can get the local space axis with getTargetVec or getUpVec etc..

In principle it should play no role whether the quaternion is for a game entity or the camera, as long as the camera is used like an object in the world. But exactly this is not clear from what the OP shows. My current problems in understanding it are the following:

According to this line

screenSpace = projMat * camMat * translateMat * rotateMat * scaleMat * positionVec

I assume that you're using column vector matrices. The quaternion-to-matrix conversion sets the rows of the matrix to the side, up, and forward vectors.

retMat.setRow(NSVec4Df(getRightVec(), 0.0f), 0);
retMat.setRow(NSVec4Df(getUpVec(), 0.0f), 1);
retMat.setRow(NSVec4Df(getTargetVec(), 0.0f), 2);

This hints at the usage of row vector matrices. Are you sure that you use it correctly?

Moreover, in

screenSpace = projMat * camMat * translateMat * rotateMat * scaleMat * positionVec

the camMat need to be the viewMat == the inverse camMat. Have you considered this? The following line

camMat = rotateMat * translateMat * camOrigin

let's me assume that camMat should in fact be the viewMat (it is computed in reverse order), but it isn't clear whether you also invert rotateMat and translateMat accordingly, because what you actually need to apply is the rule

( T * R )-1 = R-1 * T-1

The fact that the inverse rotation is identical to the transposed rotation may explain why you set the row vectors in the matrices instead of the column vectors. However, the quaternion class is not especially for the camera but for general use. As such you should not have hidden side effects in it. Moreover, there is still nothing said about the inversion of the translation.

Even if you actually do all this mathematically right (I haven't checked the formulas whether they match), naming it the way you're doing it is confusing, because you violate a common naming convention. As you can see from my post, not following the convention brings up many questions just to make sure we're speaking about the same.

I assume that you're using column vector matrices. The quaternion-to-matrix conversion sets the rows of the matrix to the side, up, and forward vectors.

retMat.setRow(NSVec4Df(getRightVec(), 0.0f), 0);
retMat.setRow(NSVec4Df(getUpVec(), 0.0f), 1);
retMat.setRow(NSVec4Df(getTargetVec(), 0.0f), 2);

This hints at the usage of row vector matrices. Are you sure that you use it correctly?

I am using row vector matrices - In the line there I set the first, second, and third rows of the matrix to the right, up, and target vectors respectively. This is how you are supposed to build the rotation matrix for a quaternion isnt it? Am I supposed to be setting the columns of the matrix rather than the rows?

You are correct about camMat needing to be viewMat - but I wasn't aware that I needed to invert the rotation and translation matrices before multiplying them together.. That might explain some issues - the strange thing is that it has worked this way for a long time now and only when I decided to switch things around to Quaternions has it given me trouble..

I am using row vector matrices ...

Are you sure? Because if you use row vectors, then you need to compute

screenSpace = positionVec * scaleMat * rotateMat * translateMat * viewMat * projMat  

instead. Perhaps you are confusing the meaning of "row / column vectors" and the memory layout "row / column major"? The former one is based on the mathematical prescription of how to compute a matrix product, while the latter one means how the 2D arrangement of elements in a matrix are mapped into the 1D computer memory.

… In the line there I set the first, second, and third rows of the matrix to the right, up, and target vectors respectively. This is how you are supposed to build the rotation matrix for a quaternion isnt it? Am I supposed to be setting the columns of the matrix rather than the rows?

The names side (or right), up, and forward (or target, in your case) vectors are used for specific directions, in particular the principal positive axes of a local co-ordinate system. They are to be set as columns if using column vectors, and they need to be set as rows if using row vectors. Not doing so so means to store in fact the transpose and hence inverse rotation. Internally the setRow and setColumn routines need to consider whether row major or else column major storage convention is chosen, and store the values accordingly. Not doing it correctly again means a transpose and hence inverse rotation.

the strange thing is that it has worked this way for a long time now and only when I decided to switch things around to Quaternions has it given me trouble..

It is important to go disciplined through this stuff, or else things get messy perhaps elsewhere. Mistakes need not be visible immediately.

Are you sure? Because if you use row vectors, then you need to compute

screenSpace = positionVec * scaleMat * rotateMat * translateMat * viewMat * projMat

So.. I am still not 100 percent sure I know which terminology I should be using - but judging by this I'm guessing I should be using the term column vector for my matrices.. After taking your advice - I changed my viewMat to

viewMat = rotateMatTranspose * translateMatInverse * camOrigin

where rotateMatTranspose is created from a Quaternion describing the camera's rotation and translation mat inverse is created from negative of the camera's position relative to its origin... And I then changed my regular Quat rotation transform code to

NSMatrix4Df NSQuaternion::getRotationMatrix() const
{
	NSMatrix4Df retMat;
	retMat.setColumn(NSVec4Df(getRightVec(), 0.0f), 0);
	retMat.setColumn(NSVec4Df(getUpVec(), 0.0f), 1);
	retMat.setColumn(NSVec4Df(getTargetVec(), 0.0f), 2);
	return retMat;
}

which means my rotation transpose matrix is formed with the following:

NSMatrix4Df NSQuaternion::getRotationTransposeMatrix() const
{
    NSMatrix4Df retMat;
    retMat.setRow(NSVec4Df(getRightVec(), 0.0f), 0);
    retMat.setRow(NSVec4Df(getUpVec(), 0.0f), 1);
    retMat.setRow(NSVec4Df(getTargetVec(), 0.0f), 2);
    return retMat;
}

And my transform pipeline for game objects is given by:

mTransform = projMat * viewMat * translateMat * rotateMat * scaleMat * positionVec;

And finally - to rotate a quaternion by an axis and an angle I use the following

void NSQuaternion::rotate(const NSVec3Df & pAxis, float pAngle)
{
	NSQuaternion localRotation;

	float halfAngle = sinf(DegreesToRadians((pAngle/2.0f)));
	localRotation.x = pAxis.x * halfAngle;
	localRotation.y = pAxis.y * halfAngle;
	localRotation.z = pAxis.z * halfAngle;
	localRotation.w = cosf(DegreesToRadians(pAngle / 2.0f));

	(*this) = (*this) * localRotation;
	normalize();
}

I am very very happy to say that taking your advice worked and completely fixed my problems - now I can use a quaternion in general to represent an objects rotation or the camera's rotation - doesn't matter.. The forward, up, and side vectors are now all pointing correctly and objects rotate how I expect them to.

Thanks so much! +1 for you

I am very very happy to say that taking your advice worked and completely fixed my problems - now I can use a quaternion in general to represent an objects rotation or the camera's rotation - doesn't matter.. The forward, up, and side vectors are now all pointing correctly and objects rotate how I expect them to.

Hurray!

So.. I am still not 100 percent sure I know which terminology I should be using

Mathematic defines that in a matrix product (look at vectors as a matrix where one of the both dimensions is set to length 1) the count of columns on the left matrix must be equal to the count of rows on the right. Hence you can write a matrix product of a 4x4 matrix and a 4 element vector either so

| a  e  i  m |   | q |
| b  f  j  n | * | r |
| c  g  k  o |   | s |
| d  h  l  p |   | t |

or else so

                 | a  b  c  d |
| q  r  s  t | * | e  f  g  h |
                 | i  j  k  l |
                 | m  n  o  p |

Notice that in the 1st solution the vector is on the right (typical for e.g. OpenGL), and it is written as a column. So we multiply 4 columns (in the matrix) by 4 rows (in the vector), which is mathematically okay. This is the use of column vectors, as you did.

Notice that in the 2nd solution the vector is on the left (typical for e.g. D3D), and it is written as a row. So we multiply 4 columns (in the vector) by 4 rows (in the matrix), which is mathematically okay. This is the use of row vectors.