I need someone to help me understand concepts around this topic.
I have read the documentation but not quite get it.
D3DRS_STENCILFUNC
The comparison function is used to compare the reference value to a stencil buffer entry.
This comparison applies only to the bits in the reference value and stencil buffer entry that are set in the stencil mask (set by the D3DRS_STENCILMASK render state).
If TRUE, the stencil test passes.
[/quote]
Let me go step by step.
1.
First i use Clear function to clear BB & DS surface with stencil of 0.
Then i draw some object with:
D3DDevice->SetRenderState(D3DRS_STENCILENABLE, TRUE);
D3DDevice->SetRenderState(D3DRS_STENCILREF, 1);
D3DDevice->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
D3DDevice->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_REPLACE);
D3DDevice->SetRenderState(D3DRS_STENCILZFAIL, D3DSTENCILOP_KEEP);
D3DDevice->SetRenderState(D3DRS_STENCILREF, 1);// this sets value of 1 to stencil where some object is drawn? Right?
sphere1->Draw();
Then i draw some other object overlaping first one with:
D3DDevice->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_EQUAL);
D3DDevice->SetRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_REPLACE);
D3DDevice->SetRenderState(D3DRS_STENCILZFAIL, D3DSTENCILOP_KEEP);
D3DDevice->SetRenderState(D3DRS_STENCILREF, 1);// value to compare with previous one? Also writes this one to stencil?
sphere2->Draw();
2. Considering D3DRS_STENCILZFAIL, is this the case (where yellow color is drawn) in this picture?
3. About D3DRS_STENCILMASK and D3DRS_STENCILWRITEMASK, how is this used? What is it "masking"/"write-masking"? I don't understand.
Can you please show me some example pseudo code?
You're a bit vague about exactly what order the stuff in your drawing is drawn in, but I think you've got essentially the right idea. Just keep in mind that stencil ref doesn't do anything by itself, it just feeds the other operations. Your first block of states will set the stencil to 1 only where stencil and depth test pass. In the second block, you tell it to set the stencil to 1 again when both tests pass, which implies that stencil was already 1. So the second draw call will never change the stencil value. You also don't seem to be setting the stencil fail value, I don't know if that's intentional or not.
Check this article and see if it helps clarify the masks. It basically does a bitwise operation on the stencil values to determine which bits are significant and which are ignored for the rest of the stencil operation.
SlimDX | Ventspace Blog | Twitter | Diverse teams make better games. I am currently hiring capable C++ engine developers in Baltimore, MD.
1.
Ok. I get this one:
(ref & mask) ComparisonOperation (oldValue & mask)
But what about D3DRS_STENCILWRITEMASK? I don't see where it is used in this expression?
Docs says its default value is 0xFFFFFFFF same as mask.
2. From article you pointed:
You can write your own formula for the value you want written into the stencil buffer as shown in the following example.
NewStencilBufferValue = (StencilBufferValue & ~StencilWriteMask) |
(StencilWriteMask & StencilOp(StencilBufferValue))
[/quote]
How to set render states to get this formula?
This makes my brain hurt.
3.
You're a bit vague about exactly what order the stuff in your drawing is drawn in...
[/quote]
First i want to understand this a bit better as i wish later to do some decals/splatting on walls. So it means i first draw walls and then spheres to tag pixels that "intersect".
Basically what it boils down to is the following code:
if (<Boolean epression setup by D3DRS_STENCILFUNC>)
{
//Here you specify what operation to do when the stencil test passes
stencilValue = <Equation setup by D3DRS_STENCILPASS>;
}
else
{
//Stencil test fails
stencilValue = <Equation setup by D3DRS_STENCILFAIL>;
}
I explain this with a code example as this is what the hardware is actually executing and I find this easier to understand then just looking at the API calls.
The masks involved in these equations are only of interest if you want to only look at say the first few bits of the actual stencil value or write to those bits. So when these are set to 0xFFFFFFFF you will write and read from the whole stencil value, when set to 0xFF000000 you will only look at/write to the first 16 bits of the stencil value (this is a 32bit value, it gets converted to 8 bits later on I believe). Bear in mind that the read and write mask don't need to have the same value.
The actual stencil test equation is just a comparison and D3DRS_STENCILFUNC sets which type of boolean expression is used to do the stencil test: "<", ">", "=", etc.
Then stencil D3DRS_STENCILFAIL or D3DRS_STENCILPASS defines what operation is done to the stencilValue: "keep", "+1", "-1", etc when the test passes or fails.
I have another related problem. I tried to reder fullscreen quad for directional light, but i want to affect everything except sky, here is my setup (but its wrong):
// ---------------------- Draw g-buffer
setRenderState(D3DRS_STENCILENABLE, TRUE);
setRenderState(D3DRS_STENCILFUNC, D3DCMP_ALWAYS);
setRenderState(D3DRS_STENCILFAIL, D3DSTENCILOP_KEEP);
setRenderState(D3DRS_STENCILZFAIL, D3DSTENCILOP_KEEP);
setRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_REPLACE);
setRenderState(D3DRS_STENCILREF, 5);
setRenderTarget(diffuseRT);
setRenderTarget(positionRT);
setRenderTarget(normalRT);
drawToGBuffer();
setRenderTarget(1, nullptr);
setRenderTarget(2, nullptr);
// ---------------------- Draw fsquad for dir light
setRenderState(D3DRS_STENCILFUNC, D3DCMP_EQUAL);
setRenderState(D3DRS_STENCILFAIL, D3DSTENCILOP_KEEP);
setRenderState(D3DRS_STENCILZFAIL, D3DSTENCILOP_KEEP);
setRenderState(D3DRS_STENCILPASS, D3DSTENCILOP_REPLACE);// If equal to gbuffer (5) draw to it, else keep? But its not?
setRenderState(D3DRS_STENCILREF, 5);
setTexture(positionRT);
setTexture(normalRT);
drawFullscreenQuad();// dir light
