Hi, been using the site and forum for a while but first post so, hello :) I have a question reguarding back face culling when using a HLSL fx file for vertex/pixel shaders. I know I can use the CULLMODE = CW or CCW function in the technique (which works), but I'm wanting to create my own function. I understand about transforming the positions of the veticies, getting their normal and dotting this against the eye vector to see if its front/back facing, what I don't understand is what to do when you have found this out. I have the following structs being used as the inputs to the vertex (a2v) and pixel (v2p):

// Structs
// Application to Vertex Shader
struct a2v
{
	float4 pos			: POSITION;
	float3 normal			: NORMAL;
	float2 texCoord			: TEXCOORD0;
	float3 tangent			: TEXCOORD1;
};

// Vertex to Pixel Shader
struct v2p
{
	float4 pos			: POSITION;
	float3 normal			: TEXCOORD0;
	float2 texCoord			: TEXCOORD1;
	float3 light			: TEXCOORD2;
	float3 eye			: TEXCOORD3;
};

// Shaders
// Vertex Shader
v2p Vs( in a2v IN )
{...}

// Pixel Shader
float4 Ps ( in v2p IN ) : COLOR
{...}

In the vertex shader I planed to do the normal dot eye calculation, if it's front faceing use the functions I have to transform and pass to the pixel for rendering as normal. If it was back facing I was thinking about setting the v2p to NULL then in the pixel shader noticing the input was null, returning NULL also, though thinking about this it doesn't really achive much (doesn't compile anyway) after a bit more thinking and a fair bit of searching I'm still stuck. I will also be including view frustrum culling which I'm guessing will use a similar method of removing the face/part of the face which is not visiable and not passing it to the pixel shader. Any help is greatly apprichiated Thanks - Tranq

As far i understand, you cant remove vertex or faces within vertex/pixel shaders. newers API version allow a new module called "Geometry shaders" which is used for add primitives within gpu; however i am not sure if you can remove primitives.

[Edited by - tpascal on December 18, 2008 10:49:35 AM]

Thanks for the replies
I'm using shader v2.0 so the vFace idea isn't an option :( **edit (using DX 9) edit**
I have quickly tried moving the position of the vertex to the eye position, though this causes problems when only 1 of the verticies of the face is back facing (on the view edge of a cylinder/sphere for example) drawing a poly from the other vert to the eye position.
what does the CULLMODE = CCW function in the technique actually do? Looking in PIX the postVS is the same with it on and off, just that the back faces are not drawn, I guess I'm trying to recode that in essance

I shall look into the geometry shader idea and see if that will fit into the application easily. If I don't move to the geometry shader idea, sticking with the custom vertex I have now, should I think about finding the backfaces pre the shader? ie before the DrawPrimitive() function? Though guessing this would also be slow and cause massive problems if it breaks a face up.. :(

Thanks again

- Tranq

[Edited by - Tranq on December 18, 2008 11:29:45 AM]

Quote:Original post by Tranq
I'm using shader v2.0 so the vFace idea isn't an option :( **edit (using DX 9) edit**

Shader 3.0 is available in DX9. Unless you want to support ancient cards there is no reason not use it.

Quote:Original post by Matt Aufderheide

Quote:Original post by Tranq
I'm using shader v2.0 so the vFace idea isn't an option :( **edit (using DX 9) edit**

Shader 3.0 is available in DX9. Unless you want to support ancient cards there is no reason not use it.

the project requires DX9 and SM 2.0 at the moment sadly. Still haven't figured out how to do it.. is there a way to do it in SM 3.0? Might be able to up the shader model to it if it does, but the DX version can't be changed for this program.