[DX9, HLSL] Vertex Transforms

Recommended Posts

Hello, I'm attempting to use a bit of hlsl to find out which vertices of an object (and just the vertices) are under some area on the screen (say a selection box, but can be any arbitrary shape). I'm using a simple quad with an attached texture that contains all or part of the vertices of the object. Also there is the texture that has the screen area (white where the selection is, transparent where it's not). I'm extracting the vertices from the texture ok, but transforming them gives the wrong results, even though I'm using the same matrices as for when they're normally rendered (I've checked with PIX, the matrices are correct). I know I need to divide the vector by the w component but there seems to be more to do. Here's the important parts of my code, all "color" values are D3DFMT_A32B32G32R32F for the time being, but I'll be using a more compact format for the resulting "image" once it's working. There isn't a vertex shader, as it's a simple quad with transformed vertices to cover the whole render area.
// globals
// a float4x4 WorldViewProj which is the transform matrix for the actual vertices
// textures and samplers for the screen mask and the vertex data

float4 p_shader(float2 lookup : TEXCOORD0) : COLOR0
{
float4 res = (float4)0; // the return vector
float4 vert = tex2D(vertSampler, lookup);
vert.w = 1; // the w component has other data that's not important

vert = mul(vert, WorldViewProj);
vert = vert/vert.w;

res = tex2D(ScreenSampler, vert.xy);

return res;
}



Share on other sites
After worldviewproj and /w your vertices are on screen (or viewport of the active render area to be precise) if x is between -1 and 1, and y is between 1 and -1 (Y is inverted), and z is between 0 and 1.

You can create another matrix, your viewport transform, to properly move the values into pixel counts if you need it, but that's more math for each vertex on the GPU. It's probably less work to transform your selection points on the CPU into the -1..1 clip space, than it is to transform each mesh point on the GPU.