# Rejecting triangles in vertex shader

## Recommended Posts

Hi, I need to find a way to reject certain triangles in the vertex shader. For example if I find a vertex match a criteria I wish to reject it (and also the other 2 forming that triangle). Now I send the vertex far away from the camera like this
if (in.Position.y < 0.0f ) in. Position,xyz = 1000.0f ;


The problem is, that not all the verices of a triangle could have Position.y < 0.0 and if not all of the vertices arent moved away from the camera of course very long, streched and annoying triangles appear [pig] How can I reject a triangle in the vertex shader, if I find one of the verices match certain criteria ? As a side question, can I get a calculate a value in a shader program and send it back to the application ? I tried something like this :

{
}

and in the application
float f ;


but that doesn't work obviously.

##### Share on other sites
What API are you using? DX9? DX10 and above? XNA?

You can't reject triangles in the vertex shader. You COULD in the geometry shader, though. What you can do is pass your criteria along to the pixel shader use clip() ... (anything under 0 means the pixel is rejected... say clip(-1)

I am not sure how things work in XNA, but in DX9, 10, etc. you have an interface for effect variable(s) that remains CPU side, but they're paired off with variables in the shader by either name, semantic, or index. Then you can use SetFloat() (or whatever) and it will set the value in the shader, and GetFloat() will retrieve the value, I believe.

Other ways to output data from shaders include writing to buffers and textures and then calling map or lock for the GPU to access it. These buffers / textures should have been created with the flags allowing CPU access.

##### Share on other sites
A vertex shader sees only one vertex at a time, so it cannot read any data from the other vertices of the primitive it belongs to. However, geometry shader does deal with primitives (assembled triangles, lines or points) and you can choose to either pass thru, amplify or reject said primitives.

Prior to shader model 5 (D3D11 hardware), you cannot modify constant buffer contents within a pixel shader. Therefore, setting a "global" (that you can read back from the GPU) within a shader won't work with just any hardware.

SM5 introduces a concept called Unordered Access buffers to which you can arbitrarily write from within pixel shaders or compute shaders. Actually, some SM4 hardware is also capable of this, but with very limited features compared to SM5 hw.

On older hardware, you have to render your data and read the rendered result to the system memory in order to access pixel shader results with the CPU.

It is conceivable that the UA writing is implemented similarly to rendering. However, direct UA writes can bypass a lot of sub-ops related to rendering, which gives better performance.

##### Share on other sites
I think the easies sollution would be to set the Alpha-Value for those
vertices to 0, when you want to reject them.

##### Share on other sites
I think the easiest sollution would be to set the Alpha-Value for those
vertices to 0, when you want to reject them.

##### Share on other sites
A geometry shader is very easy to use too, if your hardware target can handle them.

##### Share on other sites
I found a good example for the GeoShader method:

http://wiki.gamedev.net/index.php/D3DBook:Managing_Level_of_Detail

Its right in the middle of the page :)

##### Share on other sites
A long while back I implemented a technique for rejecting triangles based on some visual information. The method was used for generating silhouette geometry to outline an object in an NPR style rendering.

The key is to modify the position of one vertex with respect to the other two vertices in the triangle such that the triangle becomes back facing. In this sense you are not really rejecting the triangle, but instead making it not affect the output rendering. There are many ways to do this, but it will likely depend on how flexible your vertex format is. In your case, it sounds like you should put all three vertex positions into each vertex. Then label one of them as the 'flyer' with a vertex attribute as well. Then in the vertex shader, you can test for the 'flyer' attribute, and if that vertex is a 'flyer' then check each of the vertices positions for the condition you mentioned above. If it fails (and you want to remove it) then use the position of the other two vertices to flip the flyer to the other side of them.

That should get the job done, but it will come with a heavy memory cost as well as some additional computation. If your geometry is small then it should work alright. However, as Niko mentioned, the Geometry Shader is the proper way to do this - if your hardware is capable, then use it!

##### Share on other sites
Thanks for the help to everyone.

I'm using DX9.0c at the moment, so no geometry shaders for me..
I think I'll go with a "mark" the vertex, pass the parameter to pixel shader and clip() or set alpha=0 approach.

## Create an account

Register a new account

• ### Forum Statistics

• Total Topics
628333
• Total Posts
2982139

• 9
• 24
• 9
• 9
• 13