Jump to content
  • Advertisement
Sign in to follow this  

Stencil buffer for masking

This topic is 3958 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I want to draw a fog/smoke effect on top of some white text (bitmap fonts), and have the fog/smoke confined within the insides of the letters themselves. Effectively, only draw the smoke effect on the letters themselves, and not the background. I gather I need to use the stencil buffer to do this, to create a mask while rendering the letters, which I can then use to control where the smoke effect is rendered. I'm a little bit lost as to how I go about this though. In my head, I have it so that it goes like this: 1) Clear the stencil buffer 2) When rendering the letters, you set values in the stencil buffer 3) Then render the smoke effect, testing and rendering the pixel only if the back buffer pixel's corresponding pixel on the stencil has a '1' value Something like that? but how do you define how values in the stencil buffer are set? I'm obviously rendering my text as textured quads, so how do you effectively say "if this pixel is white, then stick a '1' in the stencil buffer for this pixel, otherwise put a '0'" I might have this backwards, in that i set everything to '1' instead of the letters which are zero. I've been reading up on the stencil buffer and it's one aspect of direct3d that I find really confusing, so i'd appreciate any help getting my head around this. or indeed if there is a better or easier way of doing what I want to achieve.

Share this post

Link to post
Share on other sites
You have the stencil concept almost exactly right, all you need is some code to get it working. The only part you need a solution for is only set the stencil value for where the pixels of the text are white. This can be done easily if the alpha values surrounding your text are 0, in which case you can use alpha-testing to only render pixels above a certain threshold. As for how to code this, I can show you how to do it in C++ and D3D9. For other languages or D3D versions, you'll have to find someone else. [smile]

// Enable alpha-testing, set threshold at zero
d3dDevice->SetRenderState( D3DRS_ALPHAREF, 0x00 );
d3dDevice->SetRenderState( D3DRS_ALPHATESTENABLE, TRUE );

// Now enable stenciling. Have the device set the stencil value to 1
// if the pixel passes the depth test.
d3dDevice->SetRenderState( D3DRS_STENCILENABLE, TRUE );
d3dDevice->SetRenderState( D3DRS_STENCILFUNC, D3DCMP_ALWAYS );
d3dDevice->SetRenderState( D3DRS_STENCILREF, 1 );

// Render the text here

// Now set the device to cull pixels that don't match the reference value of 1
d3dDevice->SetRenderState( D3DRS_STENCILFUNC, D3DCMP_EQUAL );

// Disable alpha-testing
d3dDevice->SetRenderState( D3DRS_ALPHATESTENABLE, FALSE );

// Render the smoke here

Share this post

Link to post
Share on other sites
hmm can't seem to get it working. I've got the cloudy effect over the entire quad I used to render each character. I.e., where I have a word drawn on screen, I now have a horizontal rectangle of the smoke texture covering the whole lot. Which would be correct, only that It's not confining itself to the white pixels.

Just to clarify some points..

When setting my presentation parameters, I set:

d3dpp.EnableAutoDepthStencil = true;
d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;

Now, I have no need for the stencil buffer other than for this text, so just before I render my text, I do:

d3d.lpD3DDevice->Clear( 0, NULL, D3DCLEAR_STENCIL ,
D3DCOLOR_XRGB(0,0,0), 1.0f, 0 );

which will set every value in the stencil buffer to 0, is this correct?

My textures didn't actually have alpha data until this point (every alpha pixel was just set to "0xff"). The textures had black backgrounds and to remove this, I just did

d3d.lpD3DDevice->SetRenderState(D3DRS_ALPHABLENDENABLE, TRUE);
d3d.lpD3DDevice->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_SRCALPHA);

So now i've got my texture creation function setting alpha to 0x00 if it detects a fully black pixel (0x00 or RGB(0,0,0)

So now for rendering the text, I turn on alpha testing and set it to test for 0x00, which it WONT render. Got all this ok.

then enable stenciling and this line i'm confused about:
d3dDevice->SetRenderState( D3DRS_STENCILFUNC, D3DCMP_ALWAYS );

Is this working because we're only rendering the white pixels now, we can always assume to set a stencil value of 1 for the pixel?

Now, the line:

I'm just rendering this text on a 2D menu, using pretransformed vertex data with code taking care of rendering order. Do I need to include z buffering in this stencil function? Can I just set 'D3DSTENCILOP_REPLACE' and forget about it?

Now in my smoke effect function, I set

d3d.lpD3DDevice->SetRenderState( D3DRS_STENCILFUNC, D3DCMP_EQUAL );

so where in the stencil there is a 1, we draw a pixel.

I've messed around with these values (setting 'D3DSTENCILOP_REPLACE' for one) but I seem to have one of these steps incorrect.

I'm using dx 8.1 but the code compiles fine so I'm assuming the functions haven't changed since..

Is there any alternative to the alpha testing for selecting where stencil values are placed?

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!