Jump to content
  • Advertisement
Sign in to follow this  
Mrfence97

Using the Stencil Buffer

This topic is 1333 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Currently, I'm trying to implement what I suppose is a standard use for the stencil buffer; I can't however, find any tutorials or samples describing what the D3D11_DEPTH_STENCIL_DESC parameters should be set to to carry this functionality out.

I'll describe what I want to do:

  1. Draw an object (with the usual z-buffer depth testing functionality), but also increment the value in the stencil buffer for each pixel, if the pixel is drawn.
  2. Clear the depth buffer back to 1.0, while preserving the newly written stencil buffer data (this will be done using the D3D11_CLEAR_DEPTH flag only, I imagine).
  3. Draw the 'main' scene using the newly cleared z-buffer for depth testing, but only draw pixels if the value in the stencil buffer is greater than 0 (i.e. the pixel was drawn to in the first pass).
  4. Clear the depth and stencil buffer, repeat etc.

Does anyone know how I would do this in DirectX 11? I could show the code that I have currently, if requested, but it's not working at all.

Many thanks!

Share this post


Link to post
Share on other sites
Advertisement

Can you back up a little and describe the actual effect you want to achieve?  Try to forget about how you want to do it (i.e forget about the "I want to do this with the stencil buffer" part), instead describe in fairly general terms what it is.  There may be a way of doing this that doesn't involve using the stencil buffer at all.

Share this post


Link to post
Share on other sites

I basically want a visual effect similar to the portals in 'Portal' (composting one view of the scene in another through a small window - the portal). The way I planned to do it was to draw the main scene as usual, before then rendering a small plane where the portal would be while incrementing those pixels in the stencil buffer. I would then render the scene from the point of view of the other portal, while using the data in the stencil buffer as a mask to prevent superfluous pixel calculations.

 

Hopefully that makes sense! I jumped to using the stencil buffer, because masking/compositing seems one of the main uses for it!

 

Thanks for the quick reply smile.png

Share this post


Link to post
Share on other sites

Why not render the portal view to a texture and place that texture on the surface as a decal?

 

Cheers!

Share this post


Link to post
Share on other sites

That's exactly the effect I'm looking for, but if I were to do that, I'd either have to re-render the whole scene as max resolution, or use a lower resolution that wouldn't look good close-up.

 

The idea of using the stencil buffer is so that it could be used as a mask, so I could render the scene at full resolution but only the areas that are required would actually be computed!

Share this post


Link to post
Share on other sites

Well, it is of course the worst case scenario that you'll re-render the scene - but it will happen with the stencil buffer too (the worst case scenario). With the decal texture you can still calculate / limit the viewport bounds using the decal bounds in order to save some fill rate.  

 

So, anyway, you do your normal rendering and render maybe a decal to the surface where you want the portal effect - use the decal to modify the stencil buffer (such as increment or set to a certain value). Clear zbuffer (not the stencil) and re-render the scene and reject all the pixels that don't meet the desired stencil value. 

 

One thing to remember is that typically clearing the z-buffer and not clearing the stencil isn't recommended.

 

Cheers!

Edited by kauna

Share this post


Link to post
Share on other sites

That last part of your comment is exactly what I'm trying to achieve, I can't get the D3D11_DEPTH_STENCIL_DESC parameters correct to get it to work, however! At the moment I have this:

// Depth stencil desc for decal drawing
// Create depth stencil state corresponding to step one of the article.
D3D11_DEPTH_STENCIL_DESC dsDescFirstpass;
dsDescFirstpass.DepthEnable = true;
dsDescFirstpass.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
dsDescFirstpass.DepthFunc = D3D11_COMPARISON_LESS;

// Stencil test parameters
dsDescFirstpass.StencilEnable = true;
dsDescFirstpass.StencilReadMask = 0xFF;
dsDescFirstpass.StencilWriteMask = 0xFF;

// Stencil operations if pixel is front-facing.

// Keep original value on fail.
dsDescFirstpass.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;      
dsDescFirstpass.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP; 
// Write to the stencil on pass.
dsDescFirstpass.FrontFace.StencilPassOp = D3D11_STENCIL_OP_INCR_SAT;
dsDescFirstpass.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;

// Stencil operations if pixel is back-facing.
// Since we do not care about back-facing pixels, always keep original value.
dsDescFirstpass.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
dsDescFirstpass.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
dsDescFirstpass.BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;   
dsDescFirstpass.BackFace.StencilFunc = D3D11_COMPARISON_NEVER;

//I then create the DSS, and set it using
device->OMSetDepthStencilState(dssFirstpass, 0);

//Depth stencil desc for re-drawing the main scene, using the stencil as a mask
D3D11_DEPTH_STENCIL_DESC dsDescSecondpass;

dsDescSecondpass.DepthEnable = true;
dsDescSecondpass.DepthWriteMask = D3D10_DEPTH_WRITE_MASK_ALL;
dsDescSecondpass.DepthFunc = D3D10_COMPARISON_LESS;

dsDescSecondpass.StencilEnable = true;
dsDescSecondpass.StencilReadMask = 0xFF;
dsDescSecondpass.StencilWriteMask = 0xFF;

// It does not matter what we write since we are not using the values after this step.
// In other words, we are only using the values to mask pixels.
dsDescSecondpass.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
dsDescSecondpass.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
dsDescSecondpass.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
// The stencil test passes if the passed parameter is equal to value in the buffer.
dsDescSecondpass.FrontFace.StencilFunc = D3D11_COMPARISON_NOT_EQUAL;

// Again, we do not care about back-facing pixels.
dsDescSecondpass.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;       
dsDescSecondpass.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
dsDescSecondpass.BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP; 
dsDescSecondpass.BackFace.StencilFunc = D3D11_COMPARISON_NEVER;

//I then create the DSS and set it with:
device->OMSetDepthStencilState(dssSecondpass, 0);

You can see that the reference value for both passes is set to 0, so the not equal comparison should mean that pixels that have been drawn to (so the stencil buffer is not 0) should pass the test, but the opposite happens! The pixels that have been drawn to fail, and the rest pass. If I then change from NOT_EQUAL to EQUAL, the exact same happens! Is there some glaringly obvious mistake?

 

(Also, I am clearing the stencil buffer to 0 before each frame).

 

Thanks for your help again smile.png

Share this post


Link to post
Share on other sites

In the first phase the reference value doesn't mean anything since the comparison passes always - but yes, the decal should have a non-zero stencil value, with the described stencil state - you use that only for the decal right? and you have a 3rd stencil state for typical scene rendering? 

 

Have you tried D3D11_COMPARISON_EQUAL and reference value of 1 ? 

Have you tried D3D11_COMPARISON_LESS and reference value of 0 ? 

 

Cheers!

 

[edit] You could try to visualize your stencil buffer content in order to verify that the data is what you expect it to be. Stencil buffers aren't always easy to understand :)

Edited by kauna

Share this post


Link to post
Share on other sites
Luna has a stencil sample to render a planar mirror (which is something similar to a portal), maybe this helps.

Though your setup looks good IMO, can't lay my finger on anything suspicious. I'd go with kauna's suggestion: Inspect your stencil with the graphics debugger or PIX.

Share this post


Link to post
Share on other sites

I have managed to fix the problem! It was a rather stupid mistake in the end - in-between setting the correct DSS and actually drawing the scene, I was calling a function that was resetting the bound DSS back to the 'default' state for the project, hence why none of the stenciling was working!

 

Thanks for all your help everyone smile.png.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!