# [SOLVED] Multiple masks ( with stencil ? )

This topic is 3348 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hi all. Look at this image (http://img50.imageshack.us/img50/363/texturemask.jpg): Say they're three different meshes (2D or 3D). The 3rd mesh is the "invisible" mesh that make the hole in the 2nd mesh (the green one) so user can view the background mesh. Could anyone give me an hint on how to do it in DirectX 10 (HLSL or C++) ?? I've tried tons of examples, and I'm quite sure I've to use the Stencil Buffer, but I really don't know how... Thank you. [Edited by - donlucacorleone on October 14, 2009 2:08:25 AM]

##### Share on other sites
first draw the background

after disable depth write. enable stencil test and use this for stencil config

STENCILFUNC = ALWAYS
STENCILREF = 0x01
STENCILPASS = REPLACE

(this sets the stencil bits to 1 for each pixel you draw on the screen)

draw the invisible object (I am assuming it is really invisible, if it is not change blending options not to draw it)

enable depth write, keep stencil enabled and change config to

STENCILFUNC = NOTEQUAL
STENCILPASS = KEEP
(keep other values same. this time it draws a pixel if only stencil value is not equal to 0x01, otherwise it does not draw it)

and draw the green object. disable stencil testing after that.

##### Share on other sites
Thank you shultays, it works well.

It'd be better if you could explain me the theory behind that algorithm you wrote. Also a link to some articles would be helpful.

That's because this is the simple case, but if I need to do more holes and masks I don't want to ask you every time :-)

Also, in DX10 there isn't the "STENCILREF = 0x01" parameter. What is it?!

Another note: for invisible object I set alpha = 0 in the shader. I assume it's correct. Am I right?

Thank you again for your helpful post.

##### Share on other sites

I am not sure how stencil works in dx10. for example for the first settings which we used to draw the invisible mesh

STENCILFUNC = ALWAYS
STENCILREF = 0x01
STENCILPASS = REPLACE

stencilref is a refference value which is used while stencil testing or writing something on stencil buffer

stencilfunc is testing function, it tests the current value in stencil buffer with stencilref. for example if STENCILFUNC = EQUAL, it will pass if the value in stencil buffer is equal to stencilref. here it is ALWAYS which means that stencil test will always pass and so we will always draw the pixel

stencilmask is used for masking a part of the stencil bits. in this example we use only 1 bit so it is masked by 0x01. while reading data from buffer it uses something like data_in_stencil_buffer & stencilmask

stencilwritemask is same as stencilmask, except that it is used while writing something on stencil buffer.

and finally stencilpass defines that what will stencil test do if that pixel passes the test. in this example we used replace, which means replace the stencil buffer value with stencilref (0x01 in this case)

so in this example, stencil test will always pass and so we always draw the pixel and if stencil test passes, we will set stencil buffer to 0x01 for that pixel

and in the second config, which is used while drawing the green mesh

STENCILFUNC = NOTEQUAL
STENCILPASS = KEEP

this time stencilfunc is notequal, which means stencil test will pass if stencil buffer value not equal to stencilref. stencilref is still 0x01 since we didn't change it. if stencil buffer is 0x01 it means that we used that pixel while drawing the invisible mesh so we don't draw green mesh pixel now.

and stencilpass = keep means that keep the value in the stencil buffer even if you pass the test. so we don't change the value in stencil buffer

all other variables are same as previous config, altough you can set STENCILWRITEMASK = 0x00 to make it sure that we dont change the content of stencil buffer since we don't use writing at all.

look here for more information, it shows that what you can use for stencilfunc, stencilpass etc

I am not sure why stencilref does not work in directx10, I never used it. I usually define my stencil config in shader file. you can change these in shader file like

technique test{    pass P0    {        VertexShader = compile vs_1_1 VS();        PixelShader  = compile ps_2_0 PS();        ZEnable = false;        ZWriteEnable = false;                STENCILENABLE=   TRUE;        STENCILFUNC = ALWAYS;        STENCILPASS = REPLACE;        STENCILREF=      0x01 ;        STENCILMASK=     0xFF ;        STENCILWRITEMASK=0xFF ;    }}

but I am not sure if this also works in directx10

yep, setting alpha = 0 should work

##### Share on other sites
Thank you very much for the explanation! Now it's really clear.

I did it in HLSL too.

But I'm sure that StencilRef doesn't exist in DX10. Perhaps the reference could be the second parameter in the shader function SetDepthStencilState(StencilMask_MyMask, 1) ??

Or what else?! Maybe the pixel alreay written in the Stencil Buffer !?

If someone else (I know you, shultays, are not working in DX10) could clarify it, I'd really appreciate.

Thanks.

##### Share on other sites
I've to correct myself:
the StencilRef value you used with DX9 is the second parameter of the DX10 SetDepthStencilState(StencilMask_MyMask, this_is_the_stencil_ref).

Documentation is available here:
http://msdn.microsoft.com/en-us/library/ee415626%28VS.85%29.aspx
and
http://msdn.microsoft.com/en-us/library/ee418927%28VS.85%29.aspx

1. 1
2. 2
Rutin
16
3. 3
4. 4
5. 5

• 26
• 11
• 9
• 9
• 11
• ### Forum Statistics

• Total Topics
633703
• Total Posts
3013454
×