Quote:Original post by normalperson
I don't fully understand ProcessVertices() myself. I do not get back 2D screen space coords - I get back 3D coords. And together they make a slightly warped version of my original cube which is translated slightly, probably because of the camera position. It seems like the 3D coords are in projection space. When I apply the inverse projection I get a flattened version of the warped 3D cube. Nuts!

I could manually transform the cube into projection space and compare the results. I'll also start a thread tomorrow in the directx forums to find out exactly what ProcessVertices() is doing.

You won't be able to apply the inverse perspective matrix to a 3D point, so you have to make it 4D by adding a W component of 1.0. First convert X and Y from screen-space into clip-space, by dividing by the viewport width and height, and then inverting Y (Y = 1 - Y) since screen-space coordinates are inverted along the Y axis. After that, you should be able to apply the inverse perspective matrix and divide out the W to get a 3D point again, which should be in view space. That's assuming that the original 3D point maintained it's clip-space Z value. I just tested everything out on my end with a bunch of random points and a perspective transformation, and it worked.