Stencil Buffer Usage Clarification

Started by
4 comments, last by shultays 14 years, 8 months ago
Hello, please help me understand how to use the Stencil Buffer. I have tried some things and I can't seem to make it work for me at all so I've gone back to basics just to figure things out. Right now in my pixel shader i am just returning red: float4(1,0,0,1); and i have these things set for the render states set before the draw call: device->Clear(0, NULL, D3DCLEAR_STENCIL, D3DCOLOR_ARGB(0,0,0,0), 1.0, 0); device->SetRenderState(D3DRS_STENCILENABLE, true); device->SetRenderState(D3DRS_STENCILFUNC, D3DCMP_EQUAL); device->SetRenderState(D3DRS_STENCILREF, 2); So my assumption is that when i do this, i should not see any of that red from the Pixel Shader, but the entire screen is red. I can change the value for D3DRS_STENCILFUNC to anything and it will still show the red on the screen. Can anybody give me some guidance or a link to understand how to make use of the render states pertaining to the stencil buffer? Thanks
Advertisement
Are you using a depth buffer format that includes stencil bits?
here is the create call of my depth stencil buffer:
device->CreateDepthStencilSurface(screenWidth, screenHeight, D3DFMT_D24S8,
D3DMULTISAMPLE_NONE, 0, true, &dsb, NULL);
did you choose a stencil enabled format while creating d3ddev?

like

d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;

taytay
i did, here are the values:

D3DPRESENT_PARAMETERS d3dpp;
ZeroMemory(&d3dpp, sizeof(d3dpp));
d3dpp.Windowed = true;
d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD;
d3dpp.hDeviceWindow = hWnd;
d3dpp.BackBufferFormat = D3DFMT_A8R8G8B8;
d3dpp.BackBufferWidth = SCREEN_WIDTH;
d3dpp.BackBufferHeight = SCREEN_HEIGHT;
d3dpp.EnableAutoDepthStencil = TRUE;
d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;
I am prety noob at shaders (and directx actually :p). So I might misguide you, sorry.

When I use stencil buffer and shaders, I define my stencil test variables in shader. Like

[sourcecode]    pass P1    {        VertexShader = compile vs_1_1 VS();        PixelShader  = compile ps_2_0 PS2();        ZEnable = false;        ZWriteEnable = false;                STENCILENABLE=   TRUE;        STENCILFUNC = ALWAYS;        STENCILPASS = REPLACE;        STENCILREF=      0x01 ;        STENCILMASK=     0xFF ;        STENCILWRITEMASK=0xFF ;    }


Maybe device->SetRenderState variables are disabled while using shaders or something? Again I am also new so.. =)
taytay

This topic is closed to new replies.

Advertisement