Jump to content
  • Advertisement
Sign in to follow this  

Updating the stencil buffer independently of the depth buffer

This topic is 3765 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I am attempting to write a shader that does the following (pseudocode):
if CurrentPixel passes stencil test
    write CurrentPixel.depth to depth buffer
    if CurrentPixel.alpha > AlphaClipValue
        update stencil buffer

However, I haven't been able to figure out how to update the stencil buffer conditionally while still updating the depth buffer; it seems as if it is either both or neither. I tried the following HLSL shader, and it didn't work:
HLSL Pixel Shader: Stencil test and stencil update are set by D3D calls 
on the CPU, so this shader will only execute if the stencil test passes,
and will update the stencil buffer once it finishes execution.

float ClipValue;
void Main( 
    VS_INPUT Input,
    float InDepth : TEXCOORD7,
    float4 OutColor : COLOR0,
    float OutDepth : DEPTH
    // Update the depth buffer
    OutDepth = InDepth;

    // If the alpha is clipped, cancel the shader before writing to the
    // stencil buffer, but after writing to the depth buffer.
    clip( CalculateAlpha(Input) - ClipValue );

    OutColor = 0;

What am I doing wrong? Is it even possible to write to the depth buffer without updating the stencil buffer?

Share this post

Link to post
Share on other sites
You can't do that with shaders. You have to use the ROP for that: D3DRS_STENCILPASS, D3DRS_STENCILZFAIL, D3DRS_ALPHAFUNC, etc.

Actually, I don't even think that's possible with ROPs. Alpha testing happens before depth and stencil, so if alpha test fails, depth and stencil buffers won't be updated. Perhaps you could try doing zero alpha blend instead?

Share this post

Link to post
Share on other sites
Thank you for the quick response deathkrush.

How would I set up the rendering state to do what I want? I have depth testing turned off and I want to write to the depth buffer regardless of alpha value. The only time I do not want to write to the depth buffer is when the stencil test fails.

However, I only want to update the stencil buffer if the alpha test passes. Can I use render states to achieve this?

Share this post

Link to post
Share on other sites
If depth testing is turned off, writes to the depth buffer are automatically disabled. Same with stencil testing. ROP tests are performed in this order: alpha, stencil then depth. If alpha test fails, nether depth nor stencil are updated. If stencil test fails, depth is not updated. If depth test fails, stencil can be updated conditionally.

Maybe this will work:




and then in the pixels shader output either 1.0f or 0.0f alpha. I'm not completely positive on alpha blend modes though, I always get confused with those :=)

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!