Sign in to follow this  

Stencil buffer and full screen quads

This topic is 3462 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Ah, so many questions ;-) I'm trying to use the stencil buffer to "mask out" pixels when rendering a full screen quad, but it doesn't seem to work. First, I fill the stencil buffer like this:
m_renderDevice->SetState(STATE_ZENABLE, 1);
m_renderDevice->SetState(STATE_ZWRITE_ENABLE, 1);
m_renderDevice->SetState(STATE_STENCIL_ENABLE, 0);
m_renderDevice->SetState(STATE_STENCIL_FUNC, COMPARE_ALWAYS);
m_renderDevice->SetState(STATE_STENCIL_PASS, STENCIL_REPLACE);
m_renderDevice->SetState(STATE_STENCIL_FAIL, STENCIL_REPLACE);
m_renderDevice->SetState(STATE_STENCIL_ZFAIL, STENCIL_REPLACE);
m_renderDevice->SetState(STATE_STENCIL_REF, 1);
m_renderDevice->SetState(STATE_STENCIL_MASK, 0xFF);

// Stencil buffer is cleared to "0" before rendering
RenderScene();

Nevermind the DirectX-wrapper code, everything maps directly to SetRenderState calls. Now what I think this does is write a "1" to the stencil buffer wherever there's geometry When I apply this stencil buffer when rendering "regular" geometry, it works as expected:
m_renderDevice->SetState(STATE_ZWRITE_ENABLE, 1);
m_renderDevice->SetState(STATE_STENCIL_ENABLE, 1);
m_renderDevice->SetState(STATE_STENCIL_FUNC, COMPARE_EQUAL);
m_renderDevice->SetState(STATE_STENCIL_PASS, STENCIL_KEEP);
m_renderDevice->SetState(STATE_STENCIL_FAIL, STENCIL_KEEP);
m_renderDevice->SetState(STATE_STENCIL_ZFAIL, STENCIL_KEEP);
m_renderDevice->SetState(STATE_STENCIL_REF, 0);
RenderSkybox();

This will render only the parts of the skybox where no scene geometry was rendered before. However, applying exactly the same render states when rendering a fullscreen quad results in nothing being rendered. My fullscreen quad consists of pre-transformed vertices (if that has anything to do with it). So I was wondering if there's an explanation for this, or if I just screwed up. Thanks a lot in advance!

Share this post


Link to post
Share on other sites
In the first bit of code you disable stenciling?

m_renderDevice->SetState(STATE_STENCIL_ENABLE, 0);


I would guess that this results in your stencil buffer being empty (0) after your first pass.

Share this post


Link to post
Share on other sites
I do feel a little bit stupid, but in my defense, I read in a book that you have to disable stencil-testing when writing to the stencil buffer. But now it works and so the book must be wrong.

Thanks a bunch!

Share this post


Link to post
Share on other sites
The stencilenable renderstate enables or disables stencil globally. To disable writing you use the Keep values for the stencilpass/fail states. To disable testing and always write you use Always as your stencil func. Makes sense when you think about it!

Share this post


Link to post
Share on other sites

This topic is 3462 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this