The entire bottom row is used for the perspective division, not just the B-element.I'm not really sure if A is used at all, I did this a long time ago, but B is supposed to be the perspective divide element.

Or at least B and the projection matrix modify the 4th element of the position, so that in clip space if you divide by it, you get the ndc coordinates.

For example, for an orthographic projection, the bottom row is [0, 0, 0, x] where x is some value depending on the parameters of the projection matrix. That means that the fourth component of the resulting vector after multiplication is the fourth component of the in multiplying vector, to some scale factor. For glOrtho, x=1 at all times though, which means that as long as the Z-coordinate of the input is 1.0, which is often the case, there fourth component of the resulting vector is also 1.0, and there is effectively no perspective division, hence no perspective since it is an orthographic projection.

On the other hand, for a perspective matrix, the bottom row is [0, 0, x, 0]. That means that the fourth component of the resulting vector is the third component of the multiplying vector, to some scale factor x. The third component is the depth, and hence the perspective division is now dependent on the depth; you now get a perspective effect.

Likewise, the first two elements of the bottom row can also be non-zero to get a perspective effect along the X and/or the Y-axis instead.

It doesn't make much sense to talk about how to apply these elements to a 3-element vector. You simply cannot multiply a 3-element vector by a 4-by-4 matrix in the first place. What you do when adding multiplying the 3-element vector by the rotation part and then adding the translation part as a separate step is really just assuming that the missing fourth component of the 3-element vector is unity. In order to handle the fourth row of the matrix correctly, you have to do the same assumption again, carry out the multiplication, and see how the bottom row affects the other three elements, as well as performing the final perspective division to ensure that the assumption that the fourth component really is unity even after the multiplication.how would I apply A and B to a Vec3? ( I know how to translate and rotate )

In the end, you really have to carry out a full 4-vector times 4x4-matrix multiplication, although you can assume that one element is unity and eliminate its multiplication with the corresponding elements of the matrix, and just add them.