Sign in to follow this  
ultimate-tester

2D Graphics Stencil Buffer

Recommended Posts

Hello all.

 

I have tried all day to get my stencil buffer to work for a simple 2D rectangle.

What I want to accomplish is a simple 2D filled rectangle in red and in the center a small rectangle is "taken out" so its fully transparent in the center.

Internet tells me to use a stencil buffer. Problem is, MSDN does not explain how to use it properly.

I think the problem lays somewhere else (for example my device not supporting it? I am using the DirectX simple sample just to learn about techniques) but here is my current code (I assure you I tried almost any combination of any of those values):

device->SetFVF(D3DFVF_XYZRHW | D3DFVF_DIFFUSE);
device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex)); // small rectangle in center
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_LESSEQUAL);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_ZERO);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex)); // big red rectangle

What am I doing wrong here? Thanks in advance for any help I can get.

Share this post


Link to post
Share on other sites
vinterberg    1236

On top of my head:

 

Disable writing to anything but the stencil. Clear it.

Then you draw the big red rectangle, with ALWAYS+INCRSAT. Then draw the small one using ALWAYS+DECRSAT, which should produce a stencil with a hole in it. Then use this stencil to render the big red rectangle with func=EQUAL (remember to set the stencil comparison value to 1 also), this time enable writing to screen :)

Share this post


Link to post
Share on other sites

Okay so I tried this, but it doesn't draw anything at all unfortunately:

device->SetRenderState(D3DRS_COLORWRITEENABLE, 0);
device->SetRenderState(D3DRS_ZWRITEENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_DECRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_COLORWRITEENABLE, D3DCOLORWRITEENABLE_ALPHA | D3DCOLORWRITEENABLE_BLUE | D3DCOLORWRITEENABLE_GREEN | D3DCOLORWRITEENABLE_RED);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_EQUAL);
device->SetRenderState(D3DRS_STENCILREF, 1);
device->DrawPrimitive(D3DPT_TRIANGLESTRIP, 0, 2);

Share this post


Link to post
Share on other sites

I got it working, my device didn't support stencil buffer so I did the following:

 

Creating the device:

d3dpp.EnableAutoDepthStencil = true;
d3dpp.BackBufferFormat = D3DFMT_X8R8G8B8;
d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;

And now the actual "clipping" using the stencil buffer:

device->Clear(0, nullptr, D3DCLEAR_STENCIL, D3DCOLOR_XRGB(0, 0, 255), 1.0f, 0);

device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_COLORWRITEENABLE, 0);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_NOTEQUAL);
device->SetRenderState(D3DRS_STENCILREF, 1);
device->SetRenderState(D3DRS_COLORWRITEENABLE, D3DCOLORWRITEENABLE_ALPHA | D3DCOLORWRITEENABLE_RED | D3DCOLORWRITEENABLE_GREEN | D3DCOLORWRITEENABLE_BLUE);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILENABLE, FALSE);

And I will end up with a big red rectangle with a smaller rectangle "taken out of it" so you can see right through the middle.

Share this post


Link to post
Share on other sites
LorenzoGatti    4449

Have you tried rendering your opaque rectangular donut as a plain old strip of 8 triangles? I would expect it to be slightly more efficient than using more memory for the stencil buffer and more fragments (ended early, but still processed) for overdrawing the hole region.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this