Sign in to follow this  

Updating the stencil buffer independently of the depth buffer

This topic is 3573 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I am attempting to write a shader that does the following (pseudocode):
if CurrentPixel passes stencil test
    write CurrentPixel.depth to depth buffer
    if CurrentPixel.alpha > AlphaClipValue
        update stencil buffer

However, I haven't been able to figure out how to update the stencil buffer conditionally while still updating the depth buffer; it seems as if it is either both or neither. I tried the following HLSL shader, and it didn't work:
/* 
HLSL Pixel Shader: Stencil test and stencil update are set by D3D calls 
on the CPU, so this shader will only execute if the stencil test passes,
and will update the stencil buffer once it finishes execution.
*/

float ClipValue;
void Main( 
    VS_INPUT Input,
    float InDepth : TEXCOORD7,
    float4 OutColor : COLOR0,
    float OutDepth : DEPTH
    )
{
    // Update the depth buffer
    OutDepth = InDepth;

    // If the alpha is clipped, cancel the shader before writing to the
    // stencil buffer, but after writing to the depth buffer.
    clip( CalculateAlpha(Input) - ClipValue );

    OutColor = 0;
}


What am I doing wrong? Is it even possible to write to the depth buffer without updating the stencil buffer?

Share this post


Link to post
Share on other sites
You can't do that with shaders. You have to use the ROP for that: D3DRS_STENCILPASS, D3DRS_STENCILZFAIL, D3DRS_ALPHAFUNC, etc.

Edit:
Actually, I don't even think that's possible with ROPs. Alpha testing happens before depth and stencil, so if alpha test fails, depth and stencil buffers won't be updated. Perhaps you could try doing zero alpha blend instead?

Share this post


Link to post
Share on other sites
Thank you for the quick response deathkrush.

How would I set up the rendering state to do what I want? I have depth testing turned off and I want to write to the depth buffer regardless of alpha value. The only time I do not want to write to the depth buffer is when the stencil test fails.

However, I only want to update the stencil buffer if the alpha test passes. Can I use render states to achieve this?

Share this post


Link to post
Share on other sites
If depth testing is turned off, writes to the depth buffer are automatically disabled. Same with stencil testing. ROP tests are performed in this order: alpha, stencil then depth. If alpha test fails, nether depth nor stencil are updated. If stencil test fails, depth is not updated. If depth test fails, stencil can be updated conditionally.

Maybe this will work:

D3DRS_ZENABLE = TRUE
D3DRS_ZWRITEENABLE = TRUE

D3DRS_STENCILENABLE = TRUE
D3DRS_STENCILFAIL = D3DSTENCILOP_KEEP
D3DRS_STENCILZFAIL = D3DSTENCILOP_KEEP
D3DRS_STENCILPASS = D3DSTENCILOP_REPLACE

D3DRS_ALPHABLENDENABLE = TRUE
D3DRS_ALPHATESTENABLE = FALSE
D3DRS_SRCBLEND = D3DBLEND_SRCALPHA
D3DRS_DESTBLEND = D3DBLEND_INVSRCALPHA

and then in the pixels shader output either 1.0f or 0.0f alpha. I'm not completely positive on alpha blend modes though, I always get confused with those :=)

Share this post


Link to post
Share on other sites

This topic is 3573 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this