I apologize if the title is somewhat misleading but I was unsure what to title this post. I am trying to develop somewhat of a flight simulator type camera system. Ideally, no matter what the orientation of the camera is relative to the world, if the user presses the "travel right" key, the camera should appear to travel right from their point of view (regardless of what it's orientation in world-space is). I have written some code that attempts perform an inverse transformation of directions relative to the camera back into world-space before rotating the camera. I should add that I am using the DirectXTK library for its math functions. Here is my code:

user_input_function(INPUT input_type)
{
    switch(input_type)
    {
        ....
        case MOTION_STATE_ROT_RIGHT:
            axis = __transform_local(Vector3(0, 0, 1));
            new_transform = cur_transform * Matrix::CreateFromAxisAngle(axis, -0.1f);
            this->renderer->SetTransform(TRANSFORM_VIEW,
            new_transform);
            break;
    }
}

Vector3 __transform_local(Vector3 &v)
{
	Matrix inv_view = this->renderer->GetInvTransform(TRANSFORM_VIEW);
	return Vector3::TransformNormal(v, inv_view);
}

When I run my program, the rotations produced are incorrect when attempting to rotate around an axis that is not strictly X, Y or Z (relative to the camera).

EDIT: Disclaimer - I'm still working on my first pot of coffee, so double-check my comments.

The view matrix is (or should be) comprised of the following elements:

Matrix A:

rx ux fx 0

ry uy fy 0

rz uz fz 0

-p*r -p*u -p*f 1

(using <*> to indicate a dot product>

where the camera's right vector is ( rx, ry, rz ), the camera's up vector is ( ux uy uz ), and the camera's direction is ( fx fy fz ). The camera's position is p.

More simply, if you take the inverse of the view matrix, that inverse is comprised of the row vectors:

Matrix B:

right-vector (rx, ry, rz, 0)

up-vector (ux, uy, uz, 0)

dir-vector (dx, dy, dz, 0)

cam position (px, py, pz, 1)

Assuming by "rotate" you mean to rotate about the up-vector, you can use the up-vector as the axis of rotation.

N.B., to create a proper rotation, you need to translate-to-the-origin, rotate, translate-back-to-position. And that's in world space, not view space. You'd need to reconstruct the view matrix as the inverse of the modified matrix B.

Rather than deriving camera information from the view matrix, a better approach would be to construct the view matrix from camera orientation variables. I.e., maintain camera orientation variables something like cam-pos, cam-up and cam-dir, and modify those variables to change the camera's orientation. When the camera orientation changes, construct the view matrix from Matrix::CreateLookAt( cam-pos, cam-pos + cam-dir, cam-up). If you're more comfortable with it, you can setup appropriate camera variables and use Matrix::CreateFromYawPitchRoll.