# Stencil Buffer

This topic is 1361 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hi,

Here's a greatly simplified version of what I'm trying to accomplish:

Imagine I have a gently sloping terrain (triangle strip).  At an arbitrary position on the terrain I want to put a cardboard box (with no lid).  The box will live at the base-Y position of the terrain, but the terrain might be slightly higher at this point, making the box appear to be partially buried.  This is exactly what I want.  Now imagine the camera looks down into the box.  I want to show the box contents, or simply just the bottom of the box.

I thought the best way to accomplish this would be with a stencil buffer, however I'm struggling to get it to work (following Chapter 13 in Introduction to 3D Game Programming).  Here's what I'm doing:

Step 1:  Set up the stencil buffer

   gpDeviceInterface->SetRenderState(D3DRS_STENCILENABLE, true);
gpDeviceInterface->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
gpDeviceInterface->SetRenderState(D3DRS_STENCILREF, 0x1);
gpDeviceInterface->SetRenderState(D3DRS_ZWRITEENABLE, false);
gpDeviceInterface->SetRenderState(D3DRS_ALPHABLENDENABLE, true);
gpDeviceInterface->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_ZERO);
gpDeviceInterface->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_ONE);


Step 2:  Build a polygon to represent the top plane of the box.  Render the polygon.

Step 3:  Render the terrain.


// According to Luna, the stencil buffer should be filled with 0x1 where the box "top" was
// rendered and 0x0 everywhere else.  So, LESSEQUAL should perform the stencil test on
// the entire scene/target
gpDeviceInterface->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_LESSEQUAL);

// If the test failed, replace the 0x0 with 0x1 so the terrain pixel will render
gpDeviceInterface->SetRenderState(D3DRS_STENCILFAIL, D3DSTENCILOP_REPLACE);

// If the test passed, keep the pixel
gpDeviceInterface->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_KEEP);

// If the depth test failed, the pixel is behind the stencil, so reject it
gpDeviceInterface->SetRenderState(D3DRS_STENCILZFAIL, D3DSTENCILOP_ZERO);

// Draw the terrain and reset D3DRS_STENCILENABLE...


I don't think I'm grokking Luna, because at this point the only terrain that renders is what's under the box "lid" (i.e. the stencil mask).  Can anyone spot what might be going on?

##### Share on other sites

If I understand what you intend, i.e., if the stencil buffer has 0 where you WANT to render the terrain, then the reference value 0x1 will be greater (not less than) than the stencil buffer value. As you have it, the ONLY time the test succeeds is when the ref value 0x1 is equal to the buffer value, i.e., where you've rendered the box.

Give D3DCMP_GREATER a try for the stencil func.

Edited by Buckeye

##### Share on other sites

Give D3DCMP_GREATER a try for the stencil func.

I've tried this and the opposite affect occurs.  I.e. The terrain is rendered everywhere except for the stencil mask, which is always not rendered (everything is always rendered with D3DCMP_GREATEREQUAL).  So, that would seem to point to my pass/fail/zfail settings, but changes to those do not do anything at all.

##### Share on other sites

If you want to draw where stencil is 0, use a stencil ref of 0 and a stencil func of D3DCMP_EQUAL, and the stencil test will pass causing drawing to happen.  Nothing else is needed.

The part where you're getting confused is the various stencil op functions: these are just used to specify what happens to the value in the stencil buffer under various conditions (stencil pass/stencil fail/depth fail) and don't have any effect on whether or not drawing happens: drawing happens if stencil passes irrespective of these options, and they are used for setting something different. Your initial setup is also missing a few things.

So, again with the setup.  First you clear the stencil buffer to 0, so you're going to assume that everything is to be drawn unless you prove otherwise.

Then you set stencil ref 1, stencil func ALWAYS, stencil pass REPLACE, other values are unimportant.  Now draw your "mask" geometry.  That's going to set the stencil buffer to 1 everywhere your "mask" is drawn, but leave it at 0 elsewhere.

For the second pass, you want to draw everywhere that the "mask" wasn't drawn, so set stencil ref 0, stencil func EQUAL, other values are unimportant because they control how you write to the stencil buffer and you're not writing to the stencil buffer in this pass.  Draw your other geometry and it should come out OK.

##### Share on other sites

For the second pass, you want to draw everywhere that the "mask" wasn't drawn

This isn't quite what I want.  I need to draw everywhere that the "mask" failed the depth test.  For instance, there may be terrain in between the camera and the "mask" that should be drawn.  I was assuming this is what the D3DRS_STENCILZFAIL op was for, but it doesn't seem to be working how I expected it to.

##### Share on other sites

For the second pass, you want to draw everywhere that the "mask" wasn't drawn

This isn't quite what I want.  I need to draw everywhere that the "mask" failed the depth test.  For instance, there may be terrain in between the camera and the "mask" that should be drawn.  I was assuming this is what the D3DRS_STENCILZFAIL op was for, but it doesn't seem to be working how I expected it to.

Stencil zfail is for when both depth and stencil fail, and again it doesn't affect what gets drawn but rather what happens to the values currently in the stencil buffer.

It's important to be clear here: the various stencil ops have no effect on what's drawn.

The stencil test itself is composed of a combination of (1) the value currently in the stencil buffer, (2) the reference value, (3) the stencil func value and (4) the read mask.  An object that passes the stencil test is drawn, and object that fails it is not, and these are the parameters you need to adjust to get the desired result.

At this stage I think you need to explain more clearly what it is you're trying to achieve.  Right now you're focussing more on what may actually be the wrong solution to what you want.  Don't explain it in terms of "how do I configure the stencil buffer?", explain it in terms of "I want to draw this, this and this, and have the following behaviour - what do I need to do?"

##### Share on other sites

Stencil zfail is for when both depth and stencil fail, and again it doesn't affect what gets drawn but rather what happens to the values currently in the stencil buffer.

MSDN says zfail is for when the stencil passes but depth fails.  Is this correct?

Anyways, what I want to accomplish (going back to the first post) is for the terrain pixels inside the cardboard box to be discarded.  If there are terrain pixels in between the camera and the box top, they need to be rendered (this is currently where I'm struggling).  If terrain pixels behind the box get discarded, that's fine because the box would over write them anyway.  I hope that makes things a little more clear?

• 10
• 17
• 9
• 14
• 41
• ### Forum Statistics

• Total Topics
631068
• Total Posts
2997741
×