Hello all.

I have tried all day to get my stencil buffer to work for a simple 2D rectangle.

What I want to accomplish is a simple 2D filled rectangle in red and in the center a small rectangle is "taken out" so its fully transparent in the center.

Internet tells me to use a stencil buffer. Problem is, MSDN does not explain how to use it properly.

I think the problem lays somewhere else (for example my device not supporting it? I am using the DirectX simple sample just to learn about techniques) but here is my current code (I assure you I tried almost any combination of any of those values):

device->SetFVF(D3DFVF_XYZRHW | D3DFVF_DIFFUSE);
device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex)); // small rectangle in center
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_LESSEQUAL);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_ZERO);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex)); // big red rectangle

What am I doing wrong here? Thanks in advance for any help I can get.

Okay so I tried this, but it doesn't draw anything at all unfortunately:

device->SetRenderState(D3DRS_COLORWRITEENABLE, 0);
device->SetRenderState(D3DRS_ZWRITEENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_DECRSAT);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_COLORWRITEENABLE, D3DCOLORWRITEENABLE_ALPHA | D3DCOLORWRITEENABLE_BLUE | D3DCOLORWRITEENABLE_GREEN | D3DCOLORWRITEENABLE_RED);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_EQUAL);
device->SetRenderState(D3DRS_STENCILREF, 1);
device->DrawPrimitive(D3DPT_TRIANGLESTRIP, 0, 2);

I got it working, my device didn't support stencil buffer so I did the following:

Creating the device:

d3dpp.EnableAutoDepthStencil = true;
d3dpp.BackBufferFormat = D3DFMT_X8R8G8B8;
d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;

And now the actual "clipping" using the stencil buffer:

device->Clear(0, nullptr, D3DCLEAR_STENCIL, D3DCOLOR_XRGB(0, 0, 255), 1.0f, 0);

device->SetRenderState(D3DRS_STENCILENABLE, TRUE);
device->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_INCRSAT);
device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
device->SetRenderState(D3DRS_COLORWRITEENABLE, 0);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, clearVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_NOTEQUAL);
device->SetRenderState(D3DRS_STENCILREF, 1);
device->SetRenderState(D3DRS_COLORWRITEENABLE, D3DCOLORWRITEENABLE_ALPHA | D3DCOLORWRITEENABLE_RED | D3DCOLORWRITEENABLE_GREEN | D3DCOLORWRITEENABLE_BLUE);
device->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, rectangleVertex, sizeof(RectangleVertex));

device->SetRenderState(D3DRS_STENCILENABLE, FALSE);

And I will end up with a big red rectangle with a smaller rectangle "taken out of it" so you can see right through the middle.