# Frustum Planes, have to transpose view matrix first?

This topic is 1345 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I've been implementing frustum culling.  I'm going with the method of extracting the 6 frustum planes in world space.  I am using row-major matrices.

I'm doing the following to extract planes:

ViewMatrix.Transpose();
M = ViewMatrix * ProjectionMatrix;

// Left Plane
A = M[3]  + M[0];
B = M[7]  + M[4];
C = M[11] + M[8];
D = M[15] + M[12];
(Normalize)

etc.

I then check a world-space point against each plane by testing if the dot product between the two is less than 0.  This all works, but only when I do that Transpose of the ViewMatrix first.  I don't understand why that's necessary, as no-one else seems to do that! (I only discovered that it made things work through trial and error).

My ViewMatrix comes from my camera class which is a pretty straight forward quaternion-based setup.  It's an FPS style camera: I track a Yaw and Pitch angle and a Translation vector.  I construct my ViewMatrix like this:

qYaw.FromAxisAngle(0, 1, 0, YawAngle);
qPitch.FromAxisAngle(1, 0, 0, PitchAngle);
q = qPitch * qYaw;

ViewMatrix.Identity();
ViewMatrix.Translate(Position);
ViewMatrix = (ViewMatrix * q.ToMatrix()).Inverse();

I've been reading over quaternion/matrix and frustum-plane descriptions for a few days and I can't figure out why I need to transpose that ViewMatrix before extracting the planes.  Can anyone offer my any insights?

##### Share on other sites

First of all, you need to transpose M, not only ViewMatrix.

Secondly, are you sure you need inverse matrix? If your position is zero then transpose and inverse matrices are identical, that might be source of the problem, which you "fix" by transposing.

Edited by Zaoshi Kaba

##### Share on other sites

But then why do I need to transpose M?  No tutorial on extracting the planes ever suggests doing that.  I only discovered that transposing the ViewMatrix made it work by trial and error.

And that ViewMatrix is coming from my camera class.  It calculates the camera transformation matrix, of which the View Matrix is the inverse.

##### Share on other sites

Transposing turns row-major matrix into column-major one (and vice-versa). However if you transpose only one of them you end up multiplying row-major with column-major and result is not right at very least. That's why you need to transpose M and not only ViewMatrix.

So if you need to transpose maybe your matrices are actually column-major?

I'm not good at this stuff either.

##### Share on other sites

I know my matrices are row major.  I know I don't need to transpose M.  Right now it only works when I transpose just the ViewMatrix.  I only need to do this for extracting the frustum planes; everything else (ie the whole chain of rendering) works as one would expect with row-major matrices.

I appreciate your trying to help me though!

##### Share on other sites

The way you create your ViewMatrix seems to be column major order since the rotation is on the right. In row major it should be on the left.

ViewMatrix * q.ToMatrix()

However, the way you join your view and projection matrix is row major, the way you seem to want to go.

ViewMatrix * ProjectionMatrix


Try changing your ViewMatrix code to look like this

ViewMatrix.Identity();
ViewMatrix.Translate(Position);
ViewMatrix = (q.ToMatrix() * ViewMatrix).Inverse();

Also, what doesn't seem to work about your code? There might be some clues in the way that the algorithm fails.

##### Share on other sites

if you transpose view matrix entirely, you get inverted 3x3 rotation part, what is corect, but your position part gets not inverted- it just becomes a row.

If you want to invert view matrix, than transpose the 3x3 rotation part, and multiply the position part 3 components by -1. Leave the bottom row 0,0,0,1.

You have row layout of matricies in memory, but wheather they are column or row ones, depends on how you transform vectors by them (this also implies the way you multiply matricies then, see HappyCoder post). I am sure you use column majoring, do not confuse it with layout of components.

to explain more, orthogonal matrix transpose is its inverse, but if this transposed matrix is considered in reverse majoring (column to row or viceversa)- it is the same matrix as before.

What you do in snippet is that you nearly invert view matrix and multiply it with projection matrix and compute (A,B,C,D) vector that is a (1,0,0,1) vecto transformed by M matrix.