I got it working now! There were two main problems with the code above. First the pitch/roll values were not needed to be added on every input. They should be =0.01f or =0 dependent on input. (not +=0.01f) The second problem was that I was making the mOrientMatrix = mArbRotMatrix, when I should have been multiplying them together. Though I had tried this, I finally realized that I had transposed my matrix multiplication, therefore getting incorrect values in return. Coupled with the first problem any time I tried a multiplied matrix I was getting very strange rotations. Once I fixed both of these everything fell into place. I actually ended up removing the two rotation on arbitrary axes, and used on combined matrix for rotation on x,y,z axes. I added a translation matrix and got that working as well. Finally I am able to fly around my world, even straight through the ground. I guess I am ready for some collision detection.

Thanks again for the pointing me in the right direction!

Coolness, I'm glad you got it to work!

I have a tremendous favor to ask you... Could you post the code that made it work? I'm stuck on the same thing for my own game and I can't figure out how to get the rotations to work correctly. Right now I'm getting your 'very strange rotations' (I've been stuck on that for a week), and I'm doing the yaw/pitch/roll arbitrary axis matrices while multiplying all the matrices together. Thanks for the info!

Nevermind about posting your code, I just got it to work. Thanks for the explanation, it really helped out! It's pretty cool how things go together when one thing is changed, heh! I had a few numbers completely wrong (treated a column matrix like a row matrix - ooooops!) but once I played around with it, it all worked out. Thanks for sharing your success!

Glad you got it working too!

Well, I thought everything was working right, hopefully you are not running in to this same problem I now have. Hopefully someone is still listening. Everything looks fine except for the speed. I am using a constant value for speed, but of course I want to be able to change the speed. Even with the constant speed, after about 30 seconds I notice speed building up. After about 1 minute it gets really fast and soon its uncontrollable. Maybe I still don't have something quite right.

Here is some code-

At the end of the onDrawFrame I call this sequence to create a newRotMatrix, which is called with glMultMatrixf(newRotmatrix, 0) just before I draw the ground and sky.
setmXYZrotMatrix(); //creates the rotation matrix mXYZrotmatrix

dirx = mXYZrotMatrix[8]; //get the forward vector from the rotation matrix
diry = mXYZrotMatrix[9];
dirz = mXYZrotMatrix[10];

setTransMatrix(speed, dirx, diry, dirz); //creates the translation matrix mTranslate matrix

multMatrix4x4(mXYZrotMatrix, mTranslateMatrix, newRotMatrix2); //combine the rotation and translation matrices into one transform matrix (newRotMatrix2)

multMatrix4x4(newRotMatrix, newRotMatrix2, newRotMatrix); //Apply the new transform to the current orientation



My rotation matrix and translation matrix builders:

public void setTransMatrix(double dist, double x, double y, double z) {
mTranslateMatrix[0] = 1;
mTranslateMatrix[1] = 0;
mTranslateMatrix[2] = 0;
mTranslateMatrix[3] = 0;

mTranslateMatrix[4] = 0;
mTranslateMatrix[5] = 1;
mTranslateMatrix[6] = 0;
mTranslateMatrix[7] = 0;

mTranslateMatrix[8] = 0;
mTranslateMatrix[9] = 0;
mTranslateMatrix[10] = 1;
mTranslateMatrix[11] = 0;

mTranslateMatrix[12] = (float) (mTranslateMatrix[12] + (x * dist));
mTranslateMatrix[13] = (float) (mTranslateMatrix[13] + (y * dist));
mTranslateMatrix[14] = (float) (mTranslateMatrix[14] + (z * dist));
mTranslateMatrix[15] = 1;
}
public void setmXYZrotMatrix() {
double cA = Math.cos(pitch);
double sA = Math.sin(pitch);
double cB = Math.cos(yaw);
double sB = Math.sin(yaw);
double cC = Math.cos(roll);
double sC = Math.sin(roll);

mXYZrotMatrix[0] = (float) (cB*cC);
mXYZrotMatrix[1] = (float) ((sA*sB*cC)+(cA*sC));
mXYZrotMatrix[2] = (float) ((-cA*sB*cC)+(sA*sC));
mXYZrotMatrix[3] = 0;

mXYZrotMatrix[4] = (float) (-cB*sC);
mXYZrotMatrix[5] = (float) ((-sA*sB*sC) + (cA*cC));
mXYZrotMatrix[6] = (float) ((cA*sB*sC)+(sA*cC));
mXYZrotMatrix[7] = 0;

mXYZrotMatrix[8] = (float) sB;
mXYZrotMatrix[9] = (float) (-sA*cB);
mXYZrotMatrix[10] = (float) (cA*cB);
mXYZrotMatrix[11] = 0;

mXYZrotMatrix[12] = 0;
mXYZrotMatrix[13] = 0;
mXYZrotMatrix[14] = 0;
mXYZrotMatrix[15] = 1;
}


Matrix multiplication is done here
public void multMatrix4x4(float[] mat1, float[] mat2, float[] newMat) {


newMat [0] = (mat1[0] * mat2[0]) + (mat1[1] * mat2[1]) + (mat1[2] * mat2[2]) + (mat1[3] * mat2[3]);
newMat [1] = (mat1[0] * mat2[4]) + (mat1[1] * mat2[5]) + (mat1[2] * mat2[6]) + (mat1[3] * mat2[7]);
newMat [2] = (mat1[0] * mat2[8]) + (mat1[1] * mat2[9]) + (mat1[2] * mat2[10]) + (mat1[3] * mat2[11]);
newMat [3] = (mat1[0] * mat2[12]) + (mat1[1] * mat2[13]) + (mat1[2] * mat2[14]) + (mat1[3] * mat2[15]);

newMat [4] = (mat1[4] * mat2[0]) + (mat1[5] * mat2[1]) + (mat1[6] * mat2[2]) + (mat1[7] * mat2[3]);
newMat [5] = (mat1[4] * mat2[4]) + (mat1[5] * mat2[5]) + (mat1[6] * mat2[6]) + (mat1[7] * mat2[7]);
newMat [6] = (mat1[4] * mat2[8]) + (mat1[5] * mat2[9]) + (mat1[6] * mat2[10]) + (mat1[7] * mat2[11]);
newMat [7] = (mat1[4] * mat2[12]) + (mat1[5] * mat2[13]) + (mat1[6] * mat2[14]) + (mat1[7] * mat2[15]);

newMat [8] = (mat1[8] * mat2[0]) + (mat1[9] * mat2[1]) + (mat1[10] * mat2[2]) + (mat1[11] * mat2[3]);
newMat [9] = (mat1[8] * mat2[4]) + (mat1[9] * mat2[5]) + (mat1[10] * mat2[6]) + (mat1[11] * mat2[7]);
newMat [10] = (mat1[8] * mat2[8]) + (mat1[9] * mat2[9]) + (mat1[10] * mat2[10]) + (mat1[11] * mat2[11]);
newMat [11] = (mat1[8] * mat2[12]) + (mat1[9] * mat2[13]) + (mat1[10] * mat2[14]) + (mat1[11] * mat2[15]);

newMat [12] = (mat1[12] * mat2[0]) + (mat1[13] * mat2[1]) + (mat1[14] * mat2[2]) + (mat1[15] * mat2[3]);
newMat [13] = (mat1[12] * mat2[4]) + (mat1[13] * mat2[5]) + (mat1[14] * mat2[6]) + (mat1[15] * mat2[7]);
newMat [14] = (mat1[12] * mat2[8]) + (mat1[13] * mat2[9]) + (mat1[14] * mat2[10]) + (mat1[15] * mat2[11]);
newMat [15] = (mat1[12] * mat2[12]) + (mat1[13] * mat2[13]) + (mat1[14] * mat2[14]) + (mat1[15] * mat2[15]);

}


The resulting values of my mTranslatematrix give constant values based on the speed. However, when they are combined with the current orientation they multiply slowly, but exponentially.. I logged the value of each, just to make sure I wasn't imagining something, and found the [12],[13],[14] values for the transform matrix to be constant with the speed. The values of the newRotMatrix do increase over time as I was seeing in the application. Everything I have read is telling me to do it this way, and I can't find anything on the web with a similar problem. I'll keep trying but so far everything I change makes something else stop working right.

I heard from some of the other math experts (I'm definitely not one of them, lol!) that floating-point imprecision can cause that to happen. The trick is to re-orthogonalize the rotation matrix. I'm using a different sized matrix (3x3 matrix), but you should be able to modify the below code and get it to work in your project:

// Orthogonalize the orientation matrix!
var rfx, rfy, rfz, rfMag;
var rux, ruy, ruz, ruMag;
var rx, ry, rz, rMag;

// Cross-product of right/forward vectors (store in 'up')...
rfx = matrix[5]*matrix[6] - matrix[8]*matrix[3];
rfy = matrix[8]*matrix[0] - matrix[2]*matrix[6];
rfz = matrix[2]*matrix[3] - matrix[5]*matrix[0];
rfMag = sqrt(rfx*rfx + rfy*rfy + rfz*rfz);
rfx = rfx/rfMag;
rfy = rfy/rfMag;
rfz = rfz/rfMag;

// Cross-product of right/up vectors (store in 'forward')...
rux = rfz*matrix[3] - rfy*matrix[6];
ruy = rfx*matrix[6] - rfz*matrix[0];
ruz = rfy*matrix[0] - rfx*matrix[3];
ruMag = sqrt(rux*rux + ruy*ruy + ruz*ruz);
rux = rux/ruMag;
ruy = ruy/ruMag;
ruz = ruz/ruMag;

// Normalize 'x' vector...
rMag = sqrt(matrix[0]*matrix[0] + matrix[3]*matrix[3] + matrix[6]*matrix[6]);
rx = matrix[0]/rMag;
ry = matrix[3]/rMag;
rz = matrix[6]/rMag;

// All's orthaganlized! Store new values into
// ship's 'ma' matrix...
matrix[0] = rx;
matrix[1] = rfx;
matrix[2] = rux;

matrix[3] = ry;
matrix[4] = rfy;
matrix[5] = ruy;

matrix[6] = rz;
matrix[7] = rfz;
matrix[8] = ruz;


If you plug in this code and notice nothing except your speed errors are gone, then it worked! If the visuals go all over the place, I probably got a couple of matrix elements inverted or something like that and some simple tweaking should get it to work (I translated from my own variables to something more understandable before posting the code). Anyway, I was told that this code doesn't have to be executed on every frame, so you can make a call to this code every now and then (every ten frames?) or perform some low-cost calculations to see if an axis has gone out of tolerance.

Hope that helps! I'm still pretty new to this, so please let me know if it works!

EDIT: I plugged in this code after I updated the old orientation matrix with the new values, just before I put things on the screen.

Thanks HobbyDude! I will try this when I get home. Before I left I tried removing the addition of mTranslateMatrix[12],13,14 in my setTranslateMatrix(). It seems to have slowed the speed increase, but not eliminated it. Hopefully combined with the code above I will get rid of it. I also read that this could happen and the need to reorthagonlize is not necessary on every frame, maybe every hundred or so. Some basic code can check for the values being within a tolerance. If you are within tolerance, then don't do it to save some computation time.
Thanks again, I'll update with my results later.

I will need to sit down and rewrite the reortho myself. For now it seems that my change to setTranslateMatrix() did the trick.

The new code would look like mTranslateMatrix[12] = (float) (x * dist);

The same for [13] & [14], with y,z values. (All other values are just from the identity matrix). I believe the addition I thought needed to be there is actually happening when I multiply the matrix by the current orientation.

The re-orthoganalize code doesn't work if I just plug it in. I think you may be using row major matrices, while mine are column major (openGL). I tried some simple adjustments but no luck. Like I said I'll just rewrite it, its just a matter of multiplication order and matrix order and I'm sure it'll go easier if i do it by hand myself.

Thanks again and happy coding!

Please notice that inaccuracies ever occur when concatenating enough transformations. You may consider to switch from concatenating rotation matrices to concatenating quaternions. The are less prone to failure accumulation and their re-orthonormalization (yes, just that) is much more simple. However, it comes with the costs to do a conversion.

Just to clarify: Rotation matrices need not only be "re-orthogonalized" (i.e. their vectors need to be enforced to be pairwise orthogonal) but "re-orthonormalized" (i.e. they also need to be enforced to have unit length). Only a orthonormal matrix is guaranteed to be a pure rotation matrix. The code snippet actually does a re-orthonormalization (AFAIS), and I just wanted to clarify that the term used for it is not exact.

...
The re-orthoganalize code doesn't work if I just plug it in. I think you may be using row major matrices, while mine are column major (openGL). I tried some simple adjustments but no luck. Like I said I'll just rewrite it, its just a matter of multiplication order and matrix order and I'm sure it'll go easier if i do it by hand myself.
...

Also for clarification: "Row major" and "column major" are terms to describe how the elements of the 2D matrix are arranged when stored in linear (i.e. 1D) memory. Hence you probably mean "row vector use" (e.g. D3D, like in v * M) or "column vector use" (e.g. OpenGL, like in M * v) instead, what plays a role how the terms of a matrix product are to be arranged.