Sign in to follow this  
Tranq

HLSL: Create own back face cull function

Recommended Posts

Tranq    122
Hi, been using the site and forum for a while but first post so, hello :) I have a question reguarding back face culling when using a HLSL fx file for vertex/pixel shaders. I know I can use the CULLMODE = CW or CCW function in the technique (which works), but I'm wanting to create my own function. I understand about transforming the positions of the veticies, getting their normal and dotting this against the eye vector to see if its front/back facing, what I don't understand is what to do when you have found this out. I have the following structs being used as the inputs to the vertex (a2v) and pixel (v2p):
// Structs
// Application to Vertex Shader
struct a2v
{
	float4 pos			: POSITION;
	float3 normal			: NORMAL;
	float2 texCoord			: TEXCOORD0;
	float3 tangent			: TEXCOORD1;
};

// Vertex to Pixel Shader
struct v2p
{
	float4 pos			: POSITION;
	float3 normal			: TEXCOORD0;
	float2 texCoord			: TEXCOORD1;
	float3 light			: TEXCOORD2;
	float3 eye			: TEXCOORD3;
};

// Shaders
// Vertex Shader
v2p Vs( in a2v IN )
{...}

// Pixel Shader
float4 Ps ( in v2p IN ) : COLOR
{...}
In the vertex shader I planed to do the normal dot eye calculation, if it's front faceing use the functions I have to transform and pass to the pixel for rendering as normal. If it was back facing I was thinking about setting the v2p to NULL then in the pixel shader noticing the input was null, returning NULL also, though thinking about this it doesn't really achive much (doesn't compile anyway) after a bit more thinking and a fair bit of searching I'm still stuck. I will also be including view frustrum culling which I'm guessing will use a similar method of removing the face/part of the face which is not visiable and not passing it to the pixel shader. Any help is greatly apprichiated Thanks - Tranq

Share this post


Link to post
Share on other sites
tpascal    120
As far i understand, you cant remove vertex or faces within vertex/pixel shaders. newers API version allow a new module called "Geometry shaders" which is used for add primitives within gpu; however i am not sure if you can remove primitives.

[Edited by - tpascal on December 18, 2008 10:49:35 AM]

Share this post


Link to post
Share on other sites
Martin    194
It IS possible to backface cull in the vertex shader but highly inefficient. Each vertex structure must contain the other 2 vertices the triangle will reference and if backfaced put the vertex behind the camera. The explosion in memory is incrediable though (each vert is now per triangle unique, i.e. indexing does not save you anything) and going to kill your performance, let the hardware do it. (Or use a geometry shader)

Share this post


Link to post
Share on other sites
Schrompf    1035
Check the vFace register in the pixel shader. It's positive for a front faces and negative for a backface, if I remember correctly. Check the HLSL documentation for the details. Compare to the register's value and clip() accordingly. The register is only available from shader model 3.0 upwards.

Share this post


Link to post
Share on other sites
Tranq    122
Thanks for the replies
I'm using shader v2.0 so the vFace idea isn't an option :( **edit (using DX 9) edit**
I have quickly tried moving the position of the vertex to the eye position, though this causes problems when only 1 of the verticies of the face is back facing (on the view edge of a cylinder/sphere for example) drawing a poly from the other vert to the eye position.
what does the CULLMODE = CCW function in the technique actually do? Looking in PIX the postVS is the same with it on and off, just that the back faces are not drawn, I guess I'm trying to recode that in essance

I shall look into the geometry shader idea and see if that will fit into the application easily. If I don't move to the geometry shader idea, sticking with the custom vertex I have now, should I think about finding the backfaces pre the shader? ie before the DrawPrimitive() function? Though guessing this would also be slow and cause massive problems if it breaks a face up.. :(

Thanks again

- Tranq

[Edited by - Tranq on December 18, 2008 11:29:45 AM]

Share this post


Link to post
Share on other sites
Quote:
Original post by Tranq
I'm using shader v2.0 so the vFace idea isn't an option :( **edit (using DX 9) edit**


Shader 3.0 is available in DX9. Unless you want to support ancient cards there is no reason not use it.

Share this post


Link to post
Share on other sites
Tranq    122
Quote:
Original post by Matt Aufderheide
Quote:
Original post by Tranq
I'm using shader v2.0 so the vFace idea isn't an option :( **edit (using DX 9) edit**


Shader 3.0 is available in DX9. Unless you want to support ancient cards there is no reason not use it.


the project requires DX9 and SM 2.0 at the moment sadly. Still haven't figured out how to do it.. is there a way to do it in SM 3.0? Might be able to up the shader model to it if it does, but the DX version can't be changed for this program.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this