i've met with a rather peculiar phenomena due to two things:
1) I'm getting more and more clueless about what's the problem so I've finally thought of simplifying things
2) i was wondering why i used atan2f whilst you suggested atanf.

So, rather than make the direction of all ships random, i set their directions to 0,0,1 in one pass (moving away from viewer) and 0,0,-1 in another (moving towards viewer). They of course only move towards you as long as you don't rotate the view ;)

Before changing anything to the GetMatrix function, in the first pass ships were facing the wrong way (towards viewer) but moving away from the viewer. In essence they were moving correctly but facing the wrong way. In the second pass they were pointing in the correct direction and moving in the right direction, both towards the viewer.

After changing both your functions to atan2f (which -i think- is the correct function), in the first pass the ships were exactly the same as with atanf: moving away from viewer but pointed towards viewer (incorrect). In the second pass the ships were moving towards viewer but pointing away from viewer (also incorrect).

I'll look into this more with 1,0,0 and 0,1,0 as well as a few others that should be obvious.

my apologies, here is the info on atan2f. I'm not sure but i think atanf (i'm using floats all around so no need for doubles) has a range of -pi/2 to pi/2, whilst atan2f has a range of -pi to pi (full circle).

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_crt_atan.2c_.atan2.asp

since i think it's correct, i've left it at atan2f with these tests. The only thing i've changed is the direction of the ships.

Direction:
1,0,0 moving right, pointing right. No complaints there :)
-1,0,0 moving left, pointing left
0,1,0; 0,-1,0 no ships show... i guess this is the error you mentioned
0,1000,1 (when normalised, approaches 0,1,0) ships moving up, pointing down
0,-1000,1 ships moving down, pointing up

1,1,0 moving up-right, pointing up-right
1,-1,0; -1,-1,0; -1,1,0 are all correct

1,0,1 moving far-right, pointing up... huh?
1,0,-1 moving near-right, pointing down
-1,0,-1 moving near-left, pointing down
-1,0,1 moving far-left, pointing down

0,1,1 moving far-up, pointing opposite direction (down-near)
0,1,-1; 0,-1,-1; 0,-1,1 all opposite of expected

As a final note, I seem to have completely ignored my m_Up vector, (perhaps) which has turned out some unexpected rotations. For example, when moving right the model has also been rotated 90degrees to the left around the _local_ Z axis (in other words before any other rotation). Since it's moving right (direction 1,0,0), this could also mean it's been rotated around the X axis 90degrees to the left after it's rotation around the Y axis 90 degrees to the right. ... When the only transformation that should be taking place is the one 90degrees to the right around the Y axis, at least when ignoring the Up vector.

This unexpected rotation occurred in all cases except forthe ones where m_Dir.x = 0.0f. In those 4 particular tests, for direction 0,1,1;0,1,-1; and 0,-1,1 it was correct (save for being pointed in the opposite direction), however in the 4th case where the direction was 0,-1,-1 the model was _locally_ (before any rotation) mirrored across the XZ plane, or -more likely- rotated (locally - before any rotation) around the Z axis 180 degrees.

These, to me, seems to indicate a wrong sign (or two), since in most cases the models are improperly rotated by 90 or 180 degrees. There's also a sort of 'stray' rotation that shouldn't be happening, so perhaps one of the three rotations is off by 90degrees (what comes to mind would be using a sine instead of cosine problem).

Quote:
D3DXMATRIX* D3DXMatrixMultiply(
D3DXMATRIX* pOut,
CONST D3DXMATRIX* pM1,
CONST D3DXMATRIX* pM2
);

The result represents the transformation M2 followed by the transformation M1 (Out = M1 * M2).

that to me says that
D3DXMatrixMultiply(out, M1, M2) is the same as in math out = M1 * M2.

The part "... represents the transformation M2 followed by the transformation M1..." does seem highly suspicious though, so again let me try and reverse everything.

After reversing all the multiply's they're not even moving in the right direction anymore...

I still thought it wrong, but as a test i reversed _only_ the rotation multiply's. Moving left-right they were pointing up, moving up-down they were pointing down, moving near-far they were pointing opposite of expected. When mixing XY, YZ and XZ directions i could not make out a pattern further enforcing my belief that this is the wrong way to multiply.

It was worth the shot though :)

And for good measure, current function:

that fixed the x = 0, z = 0, y = 1 or y = -1 problem marvelously. In fact, it's perfect with y = -1, but with y = 1 the model is mirrored across the XY plane (the top of the ship is facing away from the viewer whilst if the ship was rotated 90 degrees across the x axis it would be facing towards the viewer).

Likewise, when x = 1 or -1, the ship is facing away from the viewer whilst it should be facing up.
Their top is facing up correctly with z = 1 or -1; but with z (and only Z now) they are facing backwards, leading me to believe this is where the problem is at, and perhaps the only problem (let us hope). There remains the fact that the up vector isn't used but that's for later (and frankly seems extremely easy to implement).

First off, when direction was Z only the ships were flying in opposite direction. That's fixed now (apparently the models were made to point towards you... oops), but now the same problem has arisen for X direction only. So basically nothing's been fixed really, it's just the error was misperceived. To quickfix the 'model points towards you' error i first rotate around the Y axis by PI (half a circle), before doing anything else.



Now i figured since 'only Y direction' and 'only Z direction' were working perfectly, the fault had to lay with 'only X direction', and how X direction was put into rotation. So i decided to 'nullify' each z, y, and x m_Dir value and see if it has any effect. Sure enough, if x is 0.0f all rotations are done perfectly. Errors happen when x is a random value but y or z are 0.0f...

Unfortunately m_Dir.x is used in all 3 rotations...
D3DXMatrixRotationY(&matRot1, -atan2f(m_Dir.x, m_Dir.z));
D3DXMatrixRotationAxis(&matRot2, &tempVec3, atan2f(m_Dir.y, sqrt(m_Dir.x*m_Dir.x + m_Dir.z*m_Dir.z)));
D3DXMatrixRotationAxis(&matRot1, &m_Dir, atan2f(m_Dir.y, m_Dir.x));

back to the definition of atan2f:
float atan2f(float a, float b);
- If both parameters of atan2 are 0, the function returns 0.
- atan2 calculates the arctangent of a/b. atan2 is well defined for every point other than the origin, even if b equals 0 and a does not equal 0.

-atan2f(0, m_Dir.z) = arctangent(0/anything) = arctangent(0) = 0.
having X as 0 cancels the Y rotation, so this is where the fault may lay. The only thing wrong could be the angle however...
atan2f(m_Dir.y, sqrt(m_Dir.x*m_Dir.x + m_Dir.z*m_Dir.z)). Having x be 0 here has the same effect as Z being zero. Since z being zero acts normally, having X be zero can't affect the rotation.
atan2f(m_Dir.y, 0) = 'well defined', presumably 0. This is where the fault may lay, since having X be 0 (presumably) cancels the rotation.

if all the angles are correct, is it perhaps a mistake to rotate around the m_Dir vector for the Z rotation?

but why is it that then that with (and only with) m_Dir.x = 0.0f everything works perfectly?