I've been implementing frustum culling. I'm going with the method of extracting the 6 frustum planes in world space. I am using row-major matrices.

I'm doing the following to extract planes:

ViewMatrix.Transpose();
M = ViewMatrix * ProjectionMatrix;

// Left Plane
A = M[3]  + M[0]; 
B = M[7]  + M[4];
C = M[11] + M[8];
D = M[15] + M[12];
(Normalize)

etc.

qYaw.FromAxisAngle(0, 1, 0, YawAngle);
qPitch.FromAxisAngle(1, 0, 0, PitchAngle);
q = qPitch * qYaw; 

ViewMatrix.Identity();
ViewMatrix.Translate(Position);
ViewMatrix = (ViewMatrix * q.ToMatrix()).Inverse();

First of all, you need to transpose M, not only ViewMatrix.

Secondly, are you sure you need inverse matrix? If your position is zero then transpose and inverse matrices are identical, that might be source of the problem, which you "fix" by transposing.

But then why do I need to transpose M? No tutorial on extracting the planes ever suggests doing that. I only discovered that transposing the ViewMatrix made it work by trial and error.

And that ViewMatrix is coming from my camera class. It calculates the camera transformation matrix, of which the View Matrix is the inverse.

Transposing turns row-major matrix into column-major one (and vice-versa). However if you transpose only one of them you end up multiplying row-major with column-major and result is not right at very least. That's why you need to transpose M and not only ViewMatrix.

So if you need to transpose maybe your matrices are actually column-major?

I'm not good at this stuff either.

I know my matrices are row major. I know I don't need to transpose M. Right now it only works when I transpose just the ViewMatrix. I only need to do this for extracting the frustum planes; everything else (ie the whole chain of rendering) works as one would expect with row-major matrices.

I appreciate your trying to help me though!

if you transpose view matrix entirely, you get inverted 3x3 rotation part, what is corect, but your position part gets not inverted- it just becomes a row.

If you want to invert view matrix, than transpose the 3x3 rotation part, and multiply the position part 3 components by -1. Leave the bottom row 0,0,0,1.

You have row layout of matricies in memory, but wheather they are column or row ones, depends on how you transform vectors by them (this also implies the way you multiply matricies then, see HappyCoder post). I am sure you use column majoring, do not confuse it with layout of components.

to explain more, orthogonal matrix transpose is its inverse, but if this transposed matrix is considered in reverse majoring (column to row or viceversa)- it is the same matrix as before.

What you do in snippet is that you nearly invert view matrix and multiply it with projection matrix and compute (A,B,C,D) vector that is a (1,0,0,1) vecto transformed by M matrix.