Original Post
I am trying to use the Oblique Depth Projection code at http://www.terathon.com/code/oblique.php , but am having problems (using OpenGL). I am trying to render reflections in a Y=0 plane. I get the same problem whether I use Render to Texture, or just draw directly; I get the same issue with or without shaders. The clipping doesn't behave as expected - it clips objects partly in a -ve Y direction (i.e., objects towards the bottom of the screen are clipped away, which is opposite to what I want). Also it isn't fixed in world space, but moves with the camera (I am specifying the plane in camera space), and the plane appears to be tilted forward, so that objects nearer the camera are clipped first. This problem occurs both on NVIDIA 8600GT and Intel GMA 950. Everything works fine if I use the OpenGL user clip planes. Also, the z-ordering appears to be reversed - objects further away are drawn in front of those nearer. This seems to be due to my use of glScale3f(0,-1,0) for the reflection - if I don't reflect the objects, this issue is fixed, but the clipping still doesn't work properly (the reflection scale works fine with user clip planes). The code is:
// Calculate the clip-space corner point opposite the clipping plane
// as (sgn(clipPlane.x), sgn(clipPlane.y), 1, 1) and
// transform it into camera space by multiplying it
// by the inverse of the projection matrix
float q_x = (sgn(pl[0]) + matrix[8]) / matrix[0];
float q_y = (sgn(pl[1]) + matrix[9]) / matrix[5];
float q_z = -1.0F;
float q_w = (1.0F + matrix[10]) / matrix[14];
// Calculate the scaled plane vector
//Vector4D c = clipPlane * (2.0F / Dot(clipPlane, q));
float dot = q_x * pl[0] + q_y * pl[1] + q_z * pl[2] + q_w * pl[3];
float scale = 2.0 / dot;
// Replace the third row of the projection matrix
matrix[2] = pl[0] * scale;
matrix[6] = pl[1] * scale;
matrix[10] = pl[2] * scale + 1.0F;
matrix[14] = pl[3] * scale;
