Jump to content
  • Advertisement
Sign in to follow this  

[DX9, HLSL] Vertex Transforms

This topic is 3468 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello, I'm attempting to use a bit of hlsl to find out which vertices of an object (and just the vertices) are under some area on the screen (say a selection box, but can be any arbitrary shape). I'm using a simple quad with an attached texture that contains all or part of the vertices of the object. Also there is the texture that has the screen area (white where the selection is, transparent where it's not). I'm extracting the vertices from the texture ok, but transforming them gives the wrong results, even though I'm using the same matrices as for when they're normally rendered (I've checked with PIX, the matrices are correct). I know I need to divide the vector by the w component but there seems to be more to do. Here's the important parts of my code, all "color" values are D3DFMT_A32B32G32R32F for the time being, but I'll be using a more compact format for the resulting "image" once it's working. There isn't a vertex shader, as it's a simple quad with transformed vertices to cover the whole render area.
// globals
// a float4x4 WorldViewProj which is the transform matrix for the actual vertices
// textures and samplers for the screen mask and the vertex data

float4 p_shader(float2 lookup : TEXCOORD0) : COLOR0
   float4 res = (float4)0; // the return vector
   float4 vert = tex2D(vertSampler, lookup);
   vert.w = 1; // the w component has other data that's not important

   vert = mul(vert, WorldViewProj);
   vert = vert/vert.w;

   res = tex2D(ScreenSampler, vert.xy);

   return res;

Share this post

Link to post
Share on other sites
After worldviewproj and /w your vertices are on screen (or viewport of the active render area to be precise) if x is between -1 and 1, and y is between 1 and -1 (Y is inverted), and z is between 0 and 1.

You can create another matrix, your viewport transform, to properly move the values into pixel counts if you need it, but that's more math for each vertex on the GPU. It's probably less work to transform your selection points on the CPU into the -1..1 clip space, than it is to transform each mesh point on the GPU.

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!