# 2D Graphics Stencil Buffer

## Recommended Posts

Hello all.

I have tried all day to get my stencil buffer to work for a simple 2D rectangle.

What I want to accomplish is a simple 2D filled rectangle in red and in the center a small rectangle is "taken out" so its fully transparent in the center.

Internet tells me to use a stencil buffer. Problem is, MSDN does not explain how to use it properly.

I think the problem lays somewhere else (for example my device not supporting it? I am using the DirectX simple sample just to learn about techniques) but here is my current code (I assure you I tried almost any combination of any of those values):

device->SetFVF(D3DFVF_XYZRHW | D3DFVF_DIFFUSE);
device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex)); // small rectangle in center
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_LESSEQUAL);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_ZERO);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex)); // big red rectangle


What am I doing wrong here? Thanks in advance for any help I can get.

##### Share on other sites
vinterberg    1236

Disable writing to anything but the stencil. Clear it.

Then you draw the big red rectangle, with ALWAYS+INCRSAT. Then draw the small one using ALWAYS+DECRSAT, which should produce a stencil with a hole in it. Then use this stencil to render the big red rectangle with func=EQUAL (remember to set the stencil comparison value to 1 also), this time enable writing to screen :)

##### Share on other sites

Okay so I tried this, but it doesn't draw anything at all unfortunately:

device->SetRenderState(D3DRS_COLORWRITEENABLE, 0);
device->SetRenderState(D3DRS_ZWRITEENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_DECRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_COLORWRITEENABLE, D3DCOLORWRITEENABLE_ALPHA | D3DCOLORWRITEENABLE_BLUE | D3DCOLORWRITEENABLE_GREEN | D3DCOLORWRITEENABLE_RED);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_EQUAL);
device->SetRenderState(D3DRS_STENCILREF, 1);
device->DrawPrimitive(D3DPT_TRIANGLESTRIP, 0, 2);


##### Share on other sites

I got it working, my device didn't support stencil buffer so I did the following:

Creating the device:

d3dpp.EnableAutoDepthStencil = true;
d3dpp.BackBufferFormat = D3DFMT_X8R8G8B8;
d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;


And now the actual "clipping" using the stencil buffer:

device->Clear(0, nullptr, D3DCLEAR_STENCIL, D3DCOLOR_XRGB(0, 0, 255), 1.0f, 0);

device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_COLORWRITEENABLE, 0);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_NOTEQUAL);
device->SetRenderState(D3DRS_STENCILREF, 1);
device->SetRenderState(D3DRS_COLORWRITEENABLE, D3DCOLORWRITEENABLE_ALPHA | D3DCOLORWRITEENABLE_RED | D3DCOLORWRITEENABLE_GREEN | D3DCOLORWRITEENABLE_BLUE);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILENABLE, FALSE);


And I will end up with a big red rectangle with a smaller rectangle "taken out of it" so you can see right through the middle.

##### Share on other sites
LorenzoGatti    4449

Have you tried rendering your opaque rectangular donut as a plain old strip of 8 triangles? I would expect it to be slightly more efficient than using more memory for the stencil buffer and more fragments (ended early, but still processed) for overdrawing the hole region.