Ok this might be tricky to explain... I'm trying to orient the data from a magnetometer. Essentially, the magenetometer 'portrays' a vector pointing toward magnetic north and down into the Earth at an angle of roughly 60 degrees (from horizontal). Kind of like this:

\ y

\ |

\

-----x------- ->z

\

\

v

So when you're holding the sensor, you can imagine that values are 1.0 where an axis is in line with this vector (or -1.0 if it's facing away from this vector) and 0.0 when the axis is perpendicular to the 'flow' of the magnetic vector.

If I lay my sensor flat on the table with the y axis pointing directly at north, I get these values:

X = 0, Y = - 90, Z = 44. This is fine, you can imagine this in the picture above (the point 'v' is roughly this position).

If I now spin it 180 degrees on the table, I get these values:

X = 0, Y = -90, Z = -44. This is also fine, all that has happened is that in the sensor's coordinate space, the vector has flipped around. All good so far.

So what I'd like to do is apply a rotation matrix to these values which brings the vector into the vertical position. So when you spin the sensor around, the X and Z values change, but the Y value stays at 1.0. This will help with further calculations I need to do (I basically need to find the heading regardless of pitch/roll, etc).

I thought I could achieve this by creating a pitch-rotated matrix (by 60 degrees), get the inverse of that and transform my readings by that matrix and that does work but only in one orientation. If I spin the sensor around 180 degrees it's almost like the 'v' shape the two magnetic vectors have formed, has been rotated so that one of the vectors is up but the other is further toward the horizontal.

What's the correct maths for this? It's been a long time since I did any rotation stuff. I'm using DirectX and using their RotationYawPitchRoll method to get my matrix out.

You basically have a vector that always points toward a "fixed point" in space, but it's direction changes arbitrarily.

Rotating the vector into vertical position will not give you it's heading. The heading is already in the vector - it's the vector's direction.

If you want only the horizontal heading, then use only the x and z values of the vector (assuming that xz is your horizontal plane), with y=0.

However, if the vector is already vertical (x=0; y=0), then the horizontal heading will be (0, 0, 0), so in this case you can't get the horizontal heading, so maybe you need to rethink the problem, or handle this case differently.

It sounds like instead of rotating the magnetometer's vector, you want to project the vector onto the device's XZ plane. To do that, you should set the Y component to zero, and then normalize the result. This should work when the magnetometer is laid flat on the table, but if you are holding the device perpendicular to the magnetic field line you may end up with a reading like (0, 100, 0), which can't be meaningfully projected onto the XZ plane (because if you set the Y component to zero, then you get (0,0,0), which can't be normalized into a meaningful heading).

Samith explained it better than I did.

In addition, if you want to get the yaw as a value between -1.0 and 1.0, you can project both the sensor's shown vector and the aligned "magnetic north" vector (X = 0, Y = - 90, Z = 44) onto the XZ plane, then normalize them and take their dot product. The dot product will give you a value between 1 (when the two vectors overlap) and -1 (when they point in opposite directions). Is this what you need, or is it just a piece of information not related to the problem?

Unfortunately it's not possible to zero the Y and take the x and z because if you roll and pitch the device, those values change. What I essentially need to do is remove the roll and pitch from the magnetic vector which should allow me to take the x and z for heading. I can get the roll and pitch in degrees from the other two sensors (accelerometer and gyro). If I hold the sensor in line with Earth's magnetic vector (the vector pointing north and 60 degrees into the ground), then I will get such values as 0, 1, 0 or 1, 0, 0 or 0, 0, 1 or 0, 0, -1 depending on how I align the local axes of the device to the 'world' axes of the magnetic vector.

So assume I'm holding it with the Y axis of the device (up) in line with (but opposite) to Earth's magnetic vector, my readings are 0, -1, 0. This is expected and if I rotate (yaw) the device, as expected, the x and z readings remain at zero (or thereabouts) as does the y. It helps me to imagine a flow of water as the Earth's magnetic vector. If you hold a stick in that flow with the stick aligned to the flow, you'll get no resistance in the axis which are perpendicular to the flow. If you then rotate the stick so it's more perpendicular to the flow, you'll start getting resistance on all axis. You're correct, of course, in that it's impossible to tell yaw from an orientation where axis value is 1/-1and the other two are 0, but that's not a problem because I can effectively let the accelerometer and gyro take over in this orientation to give me the yaw.

If I hold the device on a flat level table with the z axis of the device in line with Earth's magnetic vector, I get 0, -90, 44 which is as you'd expect. Now, to keep things simple, let's say I pitch the device (rotate on the z axis). Because when the device is flat, it's out of alignment with Earth's magnetic vector, rotating on the x axis now pushes x slightly more toward the magnetic vector as you pitch it between 0 - 180 and slightly away from it when you go through 180-360. This was the reason for wanting to bring the device into Earth's magnetic vector's coordinate space, so when I pitch it or rotate it in x (for example), x would remain steady for this particular orientation and I'd be able to get accurate yaw from x and z.

Now obviously things get a little more complicated when you add roll into the mix, but it should be (in my head at least), a case of removing the roll, then removing the pitch and then moving into the Earth's magnetic vector coordinate space (or perhaps doing this bit first) and getting the yaw from the 'normalized' x and z. It's at this point that my head says "why are you trying to wrap your head around this rather than being outside with the 3d people".

Hmm, if you're not at the magnetic north pole, then your acceleration and magnetic vectors should be different. So you could use the opposite of the accel vector as "up" and cross that with the magnetic vector to get positive x, then cross that positive x with "up" to get positive z. Now you've constructed the world-space coordinate system in your device coordinates. You can then project the magnet vector onto that xz plane and get the yaw from there. This method should work regardless of the device's orientation so long as your magnetic field and gravity vectors aren't parallel.

It's early in the morning, but I hope that all made sense.

EDIT: I didn't think this all the way through! Obviously if you project the device's magnetic field vector into the coordinate system you constructed you'll get the same answer every time (0,0,1). But, you can take the device's forward z and represent it in the world coordinates you constructed, and get the yaw from that. If the device is being held such that it's pointing directly along the accel vector, though, you won't be able to derive a yaw, but in that situation yaw doesn't really have any meaning anyway.

Sorry! I meant you cross the magnetometers acceleration vector with the magnetometer's magnetic field vector. Both should be in the magnetometer's local coordinates. Basically why you do this to create a coordinate system representing "world space" in the magnetometer's device coordinates. Accel is up, and the device's magnetic field vector is a good approximation for positive z. Crossing them gives you positive x, and then crossing that positive x with up (accel) gives you a real positive z. This is similar to a very common technique for constructing a coordinate system for a camera where only "up" and the camera's view direction are known.

EDIT: here's some pseudo code

vec3 magnetometer_accel;  // measured gravity
vec3 magnetometer_field;  // measured field

vec3 world_up = normalize(-magnetometer_accel);
vec3 world_x = normalize(cross(world_up, normalize(magnetometer_field)));
vec3 world_z = cross(world_x, world_up);

vec3 magnetometer_z = vec3(0,0,1);
vec3 magnetometer_world_z = vec3(dot(magnetometer_z, world_x), dot(magnetometer_z, world_up), dot(magnetometer_z, world_z));

// now project onto xz plane
magnetometer_world_z.y = 0;
magnetometer_world_z = normalize(magnetometer_world_z);
float yaw = Atan2(magnetometer_world_z.x, magnetometer_world_z.z);

Might be a few errors in there but I think the general idea is solid.