I'm writing a matrix library.

In an opengl matrix, you have:

R R R T
R R R T
R R R T
A A A B

where R is rotation and T is translation.

what is A and B?

how would I apply A and B to a Vec3? ( I know how to translate and rotate )

how would I invert A and B?

I'm not really sure if A is used at all, I did this a long time ago, but B is supposed to be the perspective divide element.
Or at least B and the projection matrix modify the 4th element of the position, so that in clip space if you divide by it, you get the ndc coordinates.

The entire bottom row is used for the perspective division, not just the B-element.

For example, for an orthographic projection, the bottom row is [0, 0, 0, x] where x is some value depending on the parameters of the projection matrix. That means that the fourth component of the resulting vector after multiplication is the fourth component of the in multiplying vector, to some scale factor. For glOrtho, x=1 at all times though, which means that as long as the Z-coordinate of the input is 1.0, which is often the case, there fourth component of the resulting vector is also 1.0, and there is effectively no perspective division, hence no perspective since it is an orthographic projection.

On the other hand, for a perspective matrix, the bottom row is [0, 0, x, 0]. That means that the fourth component of the resulting vector is the third component of the multiplying vector, to some scale factor x. The third component is the depth, and hence the perspective division is now dependent on the depth; you now get a perspective effect.

Likewise, the first two elements of the bottom row can also be non-zero to get a perspective effect along the X and/or the Y-axis instead.

how would I apply A and B to a Vec3? ( I know how to translate and rotate )

It doesn't make much sense to talk about how to apply these elements to a 3-element vector. You simply cannot multiply a 3-element vector by a 4-by-4 matrix in the first place. What you do when adding multiplying the 3-element vector by the rotation part and then adding the translation part as a separate step is really just assuming that the missing fourth component of the 3-element vector is unity. In order to handle the fourth row of the matrix correctly, you have to do the same assumption again, carry out the multiplication, and see how the bottom row affects the other three elements, as well as performing the final perspective division to ensure that the assumption that the fourth component really is unity even after the multiplication.

In the end, you really have to carry out a full 4-vector times 4x4-matrix multiplication, although you can assume that one element is unity and eliminate its multiplication with the corresponding elements of the matrix, and just add them.

thank you for the insight. I'm still re-reading this a few times to let it sink in.

one problem I'm having is trying to create a frustum from a unit cube by multiplying each of the cube's vertices by the inverse of a projection matrix (as I'm told that is how it's done) so I can do basic view-frustum culling with world objects. currently, the result I'm getting is not a frustum.

okay, I think I understand everything you've said.

doing this wihtout inverting for the first try:

I've got a vector (1,2,3) and I want to multiply it by my perspective matrix:

0.75 0 0 0
0 1 0 0
0 0 -1.0001 -0.20001
0 0 -1 0

after applying rotation and translation, I have this:

0.75,2,-3.20031

and then to apply the projection division... this is where I'm not totally sure how to apply it.. but I think I would end up with a Vec4:

0.75,2,-3.20031,-1

I addressed that in my last paragraph; you have to perform a full 4-dimensional multiplication because you cannot multiply a 3-dimensional by a 4x4 matrix. You have to expand the multiplication using a full 4-dimensional vector.

Here's your matrix and vector and the product of the two.

>> M
M =
0.7500 0 0 0
0 1.0000 0 0
0 0 -1.0001 -0.2000
0 0 -1.0000 0
>> v
v =
1
2
3
1
>> p = M*v
p =
0.7500
2.0000
-3.2003
-3.0000

See how the fourth element of the vector p is -3.0? That's because your 3-dimensional vector is actually a 4-dimensional vector with an implicit element at the end with a value of 1.0.

The perspective division is just dividing all elements of the vector by the fourth element:

>> p./p(4)
ans =
-0.2500
-0.6667
1.0668
1.0000

So your resulting projected 3-dimensional vector is (-0.2500, -0.6667, 1.0668).

oh wow, that's easy. thank you very much! sometimes it takes baby-speak and sock puppets to explain things to me

I'm curious... what program are you using there in the last post? has some interesting notation

My guess would be MatLab or Octave.

My guess would be MatLab or Octave.

It's Matlab, to be specific.