Hello everybody,

I cannot get my stencil buffer to work with a texture mask. Here is the code. I am setting a white texture with a
black circle in the middle as my texture mask. After this, I draw a map, which should be stenciled to only show the
area within the black circle.

It's not working. Nothing shows up.

[source lang="cpp"]
glEnable(GL_STENCIL_TEST);

glClearStencil(0x0);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

// Draw texture here (white square with black circle in the middle)

glStencilFunc(GL_NOTEQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

// Draw map here

glDisable(GL_STENCIL_TEST);
[/source]

Thanks for any help.

How are you masking the black (or white) parts of the texture when you draw it?

I'm just binding and drawing the texture as a 2D quad. Is that what you are asking?

So you're not doing anything to ensure that only the circle, or the background, of the texture is drawn to the stencil buffer? The stencil buffer doesn't care what it is you're drawing. If you only want to fill the stencil buffer with pixels that are within the circle, then you must eliminate the background so that the background isn't drawn at all. For example, make it an alpha texture and use tha alpha test to reject the background pixels.

The texture is an alpha texture. With blended on, the circle is alpha'd out. Shouldn't the stencil buffer should have 0's for the circle and 1's for the background? The stencil op for the second pass should only draw the circle portion of the map.

Are you saying that I need a alpha test before the first pass?

Thanks.

You need the alpha test for this. As far as the stencil buffer is concerned, there are only four outcomes that it cares about. A fragment is either:

1. rejected early,
   
2. rejected by the stencil test,
   
3. rejected by the depth buffer test, or
   
4. passed by the depth buffer test.
   

Point 1 means the stencil buffer is not updated.

The three parameters to the glStencilOp function defines how the stencil buffer is updated for each of the points 2, 3 and 4. If the fragment is rejected before the stencil and the depth buffer test, the stencil buffer is untouched. So you need a way to reject a fragment before these tests take place. Once a fragment reaches the stencil test, the stencil buffer will be updated according to one of the three options.

What you have to keep in mind here is that setting a blending function and drawing a totally transparent pixels is not the same as not drawing the fragment. Drawing a transparent pixel means something is drawn, but that "something" happens to be the same as what is already there. Blending, even if the result is totally transparent, is still a write to the frame buffer and in order for that to happen, it has to pass to the stencil and the deph buffer test and thus update the stencl buffer.

To summarize, if a fragment is totally transparent, it still has to pass the stencil and the depth buffer test so that it can be blended into the frame buffer. So it really doesn't matter if it is opaque, partially transparent, or totally transparent: it has to pass all the way to the blending stage no matter what so it makes no difference to the outcome of the stencil buffer.

Got it to work with the alpha test as you suggested. Thanks for the help.

...and thanks for the detailed explanation.