I want to get two vectors in the vertex shader, one from near to far clipping plane and one from camera to near clipping plane. I keep getting wrong results and have no idea why...

In the vertex shader:

float3 frustumFarWorld = mul(FrustumFar[input.index], InvViewProj).xyz;
float3 frustumNearWorld = mul(FrustumNear[input.index], InvViewProj).xyz;
	
output.cameraToNear = frustumNearWorld - CameraPos; 
output.nearToFar = frustumFarWorld - frustumNearWorld;

Where FrustumFar and FrustumNear are matrices that hold the frustum corners. In PIX their values are:

Two things:

1. With D3D conventions, the near clip plane in normalized device coordinates is located at z=0, not at z=-1.

2. You need to perform homogeneous divide-by-w after transforming the frustum corner by the inverse view * projection matrix.

Two things:

1. With D3D conventions, the near clip plane in normalized device coordinates is located at z=0, not at z=-1.

2. You need to perform homogeneous divide-by-w after transforming the frustum corner by the inverse view * projection matrix.

I did those things and i still get wrong results

float4 frustumFarWorld = mul(FrustumFar[input.index], InvViewProj);
float4 frustumNearWorld = mul(FrustumNear[input.index], InvViewProj);
frustumFarWorld.xyz /= frustumFarWorld.w;
frustumNearWorld.xyz /= frustumNearWorld.w;

output.cameraToNear = frustumNearWorld.xyz - CameraPos; 
output.nearToFar = frustumFarWorld.xyz - frustumNearWorld.xyz;

1) As the first poster commented, the z-coordinates for your near and far planes are not correct for D3D. Search the documentation for "projection space" to learn how to fix this.

2) You do NOT need to divide by w to accomplish what you are trying to do. It's more complicated than that....

It looks like your frustum corners are specified in projection space. The terminology tends to be sloppy around this but typically "projection space" is the coordinate system a point is in after it has been multiplied by the projection matrix and had x, y and z divided by the w component. To go from projection space back into view space you need this process to happen in reverse. Basically you need to un-divide by w (not straight forward) and then multiply by the inverse projection. This isn't hard but it requires knowledge of how a projection matrix is constructed and how 4d vectors are converted into 2d points in screen space.

Suppose your projection matrix looks like this (it should for D3D):

| Sx  0  0  0 |
| 0  Sy  0  0 |
| 0   0 Sz  1 |
| 0   0 Tz  0 |

Multiplying a point by the matrix looks like this:

[x, y, z, 1] * | Sx  0  0  0 | = [Sx*x, Sy*y, Sz*z+Tz, z]
               | 0  Sy  0  0 |
               | 0   0 Sz  1 |
               | 0   0 Tz  0 |

Finally, to get to projection space we divide by w leaving us with this:

[(Sx*x)/z, (Sy*y)/z, (Sz*z+Tz)/z, 1] 

Notice that the w component went from being equal to z to being equal to 1. We lost some information at that point and regaining that information can't be accomplished by multiplying our projection space point by the inverse of the projection matrix. You need to reconstruct the original z value and then multiply by the inverse projection matrix.

Suppose a point in projection space has its z-component (this is it's depth buffer value) set to d:

d = (Sz*z+Tz)/z

We can solve this equation for z to get the original z-component:

z = Tz/(d - Sz)

This means the w-component of your near plane frustum corners should be:

Tz/(0 - Sz)

And the w-component of your far plane frustum corners should be:

 Tz/(1 - Sz)

The z-component of your near and far plane corners should be 0 and 1 respectively.

So basically multiply the frustum corners by Zf for far plane and Zn for near plane, then multiply with inverse(View * Proj)? Because the results are still wrong :/

float wN = Proj._34 / -Proj._33;	// = Zn
float wF = Proj._34 / (1 - Proj._33);	// = Zf
	
float4 frustumNearWorld = FrustumNear[input.index] * wN;
float4 frustumFarWorld = FrustumFar[input.index] * wF;
	
frustumNearWorld = mul(frustumNearWorld, InvViewProj);
frustumFarWorld = mul(frustumFarWorld, InvViewProj);

Results:

wN = 1

wF = 1000

(Multiply by w)

Ah, yeah I didn't do that quite right. First some terminology:

Clip Space is where you end up after multiplying by the projection matrix

Projection Space is where you end up after doing the divide by w on a clip space point

Let's work through an example where we go straight from view space to projection space all in one step and then reverse the process. It's actually harder to make mistakes with the math that way.

Assuming this projection matrix:

|Sx   0   0  0|
| 0  Sy   0  0|
| 0   0  Sz  1|
| 0   0  Tz  0|

Converting a view space point [x, y, z] into projection
space produces a new point [px, py, pz] where:

px = (Sx*x)    / z    (Equation 1)
py = (Sy*y)    / z    (Equation 2)
pz = (Sz*z+Tz) / z    (Equation 3)

Solving equation 3 for z:

z = Tz / (pz - Sz)    (Equation 4)

Now that we know z we can solve equation 1 and 2 for x and y:

x = (px*z) / Sx       (Equation 5)
y = (py*z) / Sy       (Equation 6)

Presto, given any projection space point [px, py, pz, 1] you
can use equation 4, 5 and 6 to convert that point into a view
space point.  To get to world space you simply multiply the
result by the inverse view matrix.

Alright i think this produced the right results.

Sounds correct to be but I would have to know the FOV and aspect ratio of your projection transform to verify. It should be noted that what you are doing is not usually done in the pixel shader or derived directly from the perspective transformation. A more common approach is to compute the frustum corners in view space (i.e. the camera is at the origin looking down the z-axis) using the same parameters used to construct the projection transform; this requires some basic trig. The corners can then be transformed into view space (or whatever other space) on the CPU once and passed to you shaders as constant data.

Oh yes of course i wasn't thinking...

Camera Position (0, 10, -5)

FOV (horizontal) = 45

Width = 1000

Height = 800

So, FOV (vertical) = 54.75

This obviously leads to (bottom left)

frustumFarWorld ( -517.8, -414.2, 995, 1.000 )

So it's wrong... again

Edit: I just realized that my matrix multiplication order is wrong.... Anyway i got the correct result (finally). I have no idea why i chose to do this instead of just calculating values in view space in the first place. I decided to do that instead so sorry for wasting your time :/

Anyway, if anyone's interested, here's the final code to convert from screen space to object space:

float4 frustumNearWorld = FrustumNear[input.index];
float4 frustumFarWorld = FrustumFar[input.index];
	
float z = Proj._34 / (frustumNearWorld.z - Proj._33);
frustumNearWorld.xyz = float3(frustumNearWorld.x * z / Proj._11, frustumNearWorld.y * z / Proj._22, z);
	
z = Proj._34 / (frustumFarWorld.z - Proj._33);
frustumFarWorld.xyz = float3(frustumFarWorld.x * z / Proj._11, frustumFarWorld.y * z / Proj._22, z);
	
frustumNearWorld = mul(InvView, frustumNearWorld);
frustumFarWorld = mul(InvView, frustumFarWorld);

Where: