Jump to content
  • Advertisement
Sign in to follow this  

Help: stencil buffer with a texture mask

This topic is 2332 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello everybody,

I cannot get my stencil buffer to work with a texture mask. Here is the code. I am setting a white texture with a
black circle in the middle as my texture mask. After this, I draw a map, which should be stenciled to only show the
area within the black circle.

It's not working. Nothing shows up.

[source lang="cpp"]

glStencilFunc(GL_ALWAYS, 0x1, 0x1);

// Draw texture here (white square with black circle in the middle)

glStencilFunc(GL_NOTEQUAL, 0x1, 0x1);

// Draw map here


Thanks for any help.

Share this post

Link to post
Share on other sites
I'm just binding and drawing the texture as a 2D quad. Is that what you are asking?

Share this post

Link to post
Share on other sites
So you're not doing anything to ensure that only the circle, or the background, of the texture is drawn to the stencil buffer? The stencil buffer doesn't care what it is you're drawing. If you only want to fill the stencil buffer with pixels that are within the circle, then you must eliminate the background so that the background isn't drawn at all. For example, make it an alpha texture and use tha alpha test to reject the background pixels. Edited by Brother Bob

Share this post

Link to post
Share on other sites
The texture is an alpha texture. With blended on, the circle is alpha'd out. Shouldn't the stencil buffer should have 0's for the circle and 1's for the background? The stencil op for the second pass should only draw the circle portion of the map.

Are you saying that I need a alpha test before the first pass?


Share this post

Link to post
Share on other sites
You need the alpha test for this. As far as the stencil buffer is concerned, there are only four outcomes that it cares about. A fragment is either:

  1. rejected early,
  2. rejected by the stencil test,
  3. rejected by the depth buffer test, or
  4. passed by the depth buffer test.

Point 1 means the stencil buffer is not updated.

The three parameters to the glStencilOp function defines how the stencil buffer is updated for each of the points 2, 3 and 4. If the fragment is rejected before the stencil and the depth buffer test, the stencil buffer is untouched. So you need a way to reject a fragment before these tests take place. Once a fragment reaches the stencil test, the stencil buffer will be updated according to one of the three options.

What you have to keep in mind here is that setting a blending function and drawing a totally transparent pixels is not the same as not drawing the fragment. Drawing a transparent pixel means something is drawn, but that "something" happens to be the same as what is already there. Blending, even if the result is totally transparent, is still a write to the frame buffer and in order for that to happen, it has to pass to the stencil and the deph buffer test and thus update the stencl buffer.

To summarize, if a fragment is totally transparent, it still has to pass the stencil and the depth buffer test so that it can be blended into the frame buffer. So it really doesn't matter if it is opaque, partially transparent, or totally transparent: it has to pass all the way to the blending stage no matter what so it makes no difference to the outcome of the stencil buffer. Edited by Brother Bob

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!