Jump to content
  • Advertisement
Sign in to follow this  
kevintlaw

Stencil Buffer Usage Clarification

This topic is 3317 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello, please help me understand how to use the Stencil Buffer. I have tried some things and I can't seem to make it work for me at all so I've gone back to basics just to figure things out. Right now in my pixel shader i am just returning red: float4(1,0,0,1); and i have these things set for the render states set before the draw call: device->Clear(0, NULL, D3DCLEAR_STENCIL, D3DCOLOR_ARGB(0,0,0,0), 1.0, 0); device->SetRenderState(D3DRS_STENCILENABLE, true); device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_EQUAL); device->SetRenderState(D3DRS_STENCILREF, 2); So my assumption is that when i do this, i should not see any of that red from the Pixel Shader, but the entire screen is red. I can change the value for D3DRS_STENCILFUNC to anything and it will still show the red on the screen. Can anybody give me some guidance or a link to understand how to make use of the render states pertaining to the stencil buffer? Thanks

Share this post


Link to post
Share on other sites
Advertisement
Are you using a depth buffer format that includes stencil bits?

Share this post


Link to post
Share on other sites
here is the create call of my depth stencil buffer:
device->CreateDepthStencilSurface(screenWidth, screenHeight, D3DFMT_D24S8,
D3DMULTISAMPLE_NONE, 0, true, &dsb, NULL);

Share this post


Link to post
Share on other sites
did you choose a stencil enabled format while creating d3ddev?

like

d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;

Share this post


Link to post
Share on other sites
i did, here are the values:

D3DPRESENT_PARAMETERS d3dpp;
ZeroMemory(&d3dpp, sizeof(d3dpp));
d3dpp.Windowed = true;
d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD;
d3dpp.hDeviceWindow = hWnd;
d3dpp.BackBufferFormat = D3DFMT_A8R8G8B8;
d3dpp.BackBufferWidth = SCREEN_WIDTH;
d3dpp.BackBufferHeight = SCREEN_HEIGHT;
d3dpp.EnableAutoDepthStencil = TRUE;
d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;

Share this post


Link to post
Share on other sites
I am prety noob at shaders (and directx actually :p). So I might misguide you, sorry.

When I use stencil buffer and shaders, I define my stencil test variables in shader. Like

[sourcecode]    pass P1
{
VertexShader = compile vs_1_1 VS();
PixelShader = compile ps_2_0 PS2();
ZEnable = false;
ZWriteEnable = false;

STENCILENABLE= TRUE;
STENCILFUNC = ALWAYS;
STENCILPASS = REPLACE;
STENCILREF= 0x01 ;
STENCILMASK= 0xFF ;
STENCILWRITEMASK=0xFF ;
}



Maybe device->SetRenderState variables are disabled while using shaders or something? Again I am also new so.. =)

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!