So I seem to have made somewhat of a mess with 3D coordinates and rotations between my code, D3D11 and Bullet, and want to try and get back to having a common understanding of the maths involved (not try this, ok wrong, try inveresing the yaw passed to an API, etc.).

How I logically have various parts of the game currently setup

• +x is east
• +y is up
• +z is north
• a yaw of 0 is facing north
• yaw goes clockwise, so north, east, south, west, north (so same as compass bearings)
• pitch, not really sure, 0 is obviously flat, from there positive seems to be down

And so with D3D11 my matrices are

XMMATRIX xmproj = XMMatrixPerspectiveFovLH(fov, aspectRatio, near, far);


XMMATRIX xmview;
xmview = XMMatrixTranslation(-x, -y, -z);
xmview *= XMMatrixRotationY(-yaw);
xmview *= XMMatrixRotationX(-pitch);

XMMATRIX xmviewProj = xmview * xmproj;

//...multiply with world matrix as needed, can also use xmproj and xmview as is for DirectX::BoundingFrustum
xmShaderTransform = XMMatrixTranspose(xmShaderTransform);
//store in constant buffer for shader output.pos = mul(input.pos, transform)

So far so good, things seem to render as expected fine, but then once looking at logic...

Vector3F Camera::getForwardVec()const
{
    auto c = std::cosf(-pitch);
    Vector3F facing(
        c * std::sinf(yaw),
        std::sinf(-pitch),
        c * std::cosf(yaw));
    return facing;
}

Vector2F Player::getMovementVec(Vector2F move)const
{
    //move is such that +z is forward, +x is strafe right
    //only rotates on XZ plane (pitch is camera only and slopes/steps handled later)
    auto c = std::cosf(-yaw);
    auto s = std::sinf(-yaw);
    return Vector2F(moveX * c - moveZ * s, moveX * s + moveZ * c);
}

...drawing some object...
btTransform transform;
motionstate.getworldTransform(transform);
XMFLOAT4X4 world;
transform.getOpenGLMatrix((float&)world.m);
XMMATRIX xmworld = XMLoadFloat4x4(&world);
XMMATRIX xmmatrix = xmworld * xmviewProj;
xmmatrix = XMMatrixTranspose(xmmatrix);
//use xmmatrix in shader

I am really not happy with having to have the -yaw and -pitch there to make those work... I also get a similar thing with my currently simple integration with bullet, e.g. I have to pass -yaw as the rotation to ghost object for my btKinematicCharacterController, yet if I use the Bullet getOpenGLMatrix to get a world matrix and then do "world * view * proj" I seem to get correct results?

Are those to be expected, I am kind of feeling I stuffed something up somewhere, especially with how I am handling yaw? But simply saying it goes the other way I am still not sure is correct (and leaves -yaw in getForwardVec instead...), since that changes the view matrix, and on the grounds that I am "rotating the world around the camera" being my understanding there, an inverse value makes sense (rotate the entire world counterclockwise rather than the screen/camera clockwise)...

I have already read those articles, they don't cover my issue in any way that I see....

• Yes D3D11 as I am using it is left-handed with row-major matrices. D3D supports right-handed as well, but some parts of DirectX math I used don't, and now a fair amount of other code is based around left-handed as well, and I also understand generally the GPU itself for D3D11 uses column-major, but it seems again DirectX Math does not, requiring that XMMatrixTranspose in my code.
• The main difference I am aware of from those articles is that the z axis is reversed. I also know that OpenGL is column major.
• As far as Bullet goes, I understand internally it should not matter. As long as all object positions, meshes, velocities, etc. are consistant. It even has options for putting say gravity on the Z axis if the XY plane is horizontal.
• I think the fact the OpenGL matrix it creates is right handed AND column-major has something to do with why I could use it "directly" like I showed.
• The only semi-relevant thing I saw to rotation was "To rotate a figure counterclockwise around the origin by some angle ..." which suggests that (regardless of left or right hand?) yaw should go counterclockwise. But simply doing that does not solve all my issues, it gets rid of the -yaw being passed to Bullet sure, but then the getForwardVec maths has -yaw instead, so at least part of my maths there must be wrong still? And is thinking that it should be counterclockwise right anyway? I cant find a solid answer, at least not in a context I am positive relates to use with D3D's left-handed row-major matrices.

Not sure if I understood your problem well but the way I retrieve information from Bullet before I generate a world matrix for rendering is that I get an object's origin & orientation as a btVector3 & btQuaternion, convert them to XMVECTORS, then use DirectXMath to generate a rotate and a translate matrix, multiply them to get my worldmatrix.

Like so:

btVector3 objectorigin = m_physicsManager->getCollisionObjectOrigin(i);
btQuaternion objectrotation = m_physicsManager->getCollisionObjectRotation(i);
XMVECTOR SIMDobjectorigin = XMVectorSet(objectorigin.getX(),objectorigin.getY(),objectorigin.getZ(),1.0f);
XMVECTOR SIMDobjectrotation = XMVectorSet(objectrotation.getX(),objectrotation.getY(),objectrotation.getZ(),objectrotation.getW());

//create a translation matrix
XMMATRIX positionMatrix = XMMatrixTranslationFromVector(position);
 
//create an orientation matrix
XMMATRIX orientationMatrix = XMMatrixRotationQuaternion(orientation);
 
//create a scaling matrix if necessary but omitted for clarity
XMMATRIX worldMatrix = scalingMatrix * orientationMatrix * positionMatrix;
 
result = Render(worldMatrix,viewMatrix, etc...);

I don't know if this is the best way to do it but I found this on my own and it works for me.