Jump to content
  • Advertisement
Sign in to follow this  

Stencils as screen-door transparency

This topic is 4113 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello, I've been trying to understand how to use stencils, but all examples I've seen appear to be much more complex than I need them to be. Maybe somebody can help me out. This is what I need: 1) I have a bitmap, say, 32x32 bits. 2) I then replicate this bitmap over the entire back buffer dimensions 3) Then, whenever a bit is set, I want the resulting bit in the back buffer to be displayed; if off, I don't want it displayed. I believe a 1-bit stencil buffer should do the trick, but I don't really know how to set this up. Could somebody kindly show me, with as much code as possible, how to do this with a stencil, if at all possible? My gut feeling tells me that a stencil is the way to do this in Direct3D, but I've never used stencils in the past. Thanks.

Share this post

Link to post
Share on other sites
3) Then, whenever a bit is set, I want the resulting bit in the back buffer to be displayed; if off, I don't want it displayed.

Well, this is what the stencil buffer does, you didnot specify what you really wanna do. Or maybe I misunderstood you =/

Share this post

Link to post
Share on other sites
Screen-door transparency is usually achieved with ALPHA TEST.

With alpha test, when you render a primitive, the alpha value of each pixel that is about to be displayed is tested against the alpha reference value (D3DRS_ALPHAREF) using a comparison function (D3DRS_ALPHAFUNC).

Any pixel which passes the alpha test (i.e. the result of the test function is TRUE) gets displayed (after stencil and Z buffer testing) and writes over whatever was previously at that position in the back buffer.

Any pixel which fails the alpha test (i.e. the result of the test function is FALSE) is discarded and instead of it you see the pixel that is already at that position in the back buffer.

The alpha value used for the alpha test is whatever is in alpha at the end of normal pixel processing, i.e. the value you last write into the alpha channel in your pixel shader or the result of the last stage of the final D3DTSS_ALPHAOP is you're using the DX7 style fixed function pipeline.

Commonly the value you output in alpha when using alpha test is the alpha channel of a texture with (for example) A=0 for see through areas and A=1 for opaque areas.

Once you have your code set up to load textures into a texture format that supports alpha, you have your texture set up ready to use, and your pixel processing pipeline is set up to output the texture alpha (you DON'T need to enable alpha blending for screen door transparency - to do so is just wasting performance), all you need to do to enable alpha testing is (for example):

dev->SetRenderState(D3DRS_ALPHAREF, 0);

Which means "for any pixels output from now until further notice, any pixels whose alpha value is greater than 0 should be displayed, but any whose alpha equals 0 shouldn't be displayed"

Stencil buffering has a similar reference value and test function to alpha testing, and a bunch of other settings for more advanced uses.

In usual operation: any pixel from a polygon being output that passes the test is written into the stencil buffer AND displayed onto the back buffer; any pixel that fails the stencil test isn't written into the stencil buffer and isn't displayed on the back buffer.

The important thing to understand about stencil testing is the comparison function compares against WHAT IS ALREADY IN THE STENCIL BUFFER. The other important thing to understand is all pixels of a polygon will be written to the stencil buffer unless they fail an earlier test such as alpha test.

So whilst you /could/ use stencil testing to achieve a per-pixel screen door effect, it would involve the use of alpha test anyway and would end up a lot more complicated than plain alpha test.

Stencil buffer is mostly for destination effects where what has been rendered previously is what affects the outcome of the test mosr. Alpha test is the opposite, what you are currently rendering is what affects the test the most.

Share this post

Link to post
Share on other sites

Thanks a million for the detailed response. Looks like ALPHA testing is the way to go, rather than stencils.

But I'm still not clear. Sorry about that... I'm just a beginner :-(.

1) From what I understood you are suggesting I take my 32x32 bit pattern and make a texture from it, probably using D3DSAMP_ADDRESSV=D3DTADDRESS_WRAP so it will get replicated rather than stretched. Could you confirm this?

2) Suppose I've done that. Now I'm about to draw a primitive, with untransformed x,y,z, diffuse RGBA color per vertex, and texture coordinates for another existing texture. If my screen-door transparency is a texture, how do I add additional texture coordinates for it? My original texture will probably be drawn with some linear interpolation, based on [0,1] values of the texture coordinates.

3) Also, the new texture for the screen-door transparency will be based on the bitmap pattern. But I don't know the final screen location of my primitive, as it is not in screen coordinates. How do I determine the texture coordinates for each vertex?

Sorry for the number of questions. Hopefully they all make sense.

Thanks again!

Share this post

Link to post
Share on other sites
1) Yep, with the WRAP texture address mode, the texture will repeat again at each integer multiple. So a square with texture coordinates of 0,0 at the top left corner and 2,1 at the bottom right corner will show the texture repeated twice in the U axis (and once in the V axis).

2) This depends on whether you're using vertex declarations or D3D7/fixed function style FVF codes. With vertex declarations, just add another coordinate anywhere in your vertex structure and add an extra D3DVERTEXELEMENT9 to the vertex declaration with a Usage of D3DDECLUSAGE_TEXCOORD and UsageIndex of 1 (assuming the first texture has a UsageIndex of 0). For old FVF style, use D3DFVF_TEX2 instead of D3DFVF_TEX1 and put the 2nd set of texture coordinates immediately after the 1st set in the vertex structure.

3) To answer this properly I need to clarify the effect you're trying to achieve, do you want:

a. a screen-door/mask pattern to be applied in 3D space (it sounds like you don't) ?

b. a screen-door/mask pattern that is in effect a 2D mask on top of the 3D rendered object **WHERE THE MASK MOVES WITH THE OBJECT** ?

c. a screen-door/mask pattern that is in effect a 2D mask on top of the 3D rendered object **WHERE THE MASK DOES NOT MOVE WITH THE OBJECT** ?

So far the method I've described fits 3a best because I was thinking in terms of 2D objects rendered with 2D masks rather than 3D objects with a 2D mask. I think I may have misinterpreted your original post if this is the case.

It is still possible to use alpha testing to achieve that. The trick is to transform the texture coordinates of the mask texture by the inverse of any camera and world transformation matrices being applied to the 3D object (to cancel out the 3D transformation).

However, if what you want is 3b or 3c, back transforming texture coordinates is overkill and the stencil buffer (3c) or destination alpha (3b) is more appropriate. But the discussion of alpha test isn't wasted because you'll likely need it to achieve the effect you want using stencil.

An overview of the stencil buffer method to achieve 2D masking of a 3D object:

0a. Remember to specify a depth stencil format that actually has some usable bits of stencil.

0b. Remember to specify D3DCLEAR_STENCIL in your clear operation and clear the reference value to 0 (you could use other reference values for other methods).

1a. Set up a textured 2D primitive, e.g. a quad/sprite for your mask, filling the whole screen and using texture repeats if necessary. This will have your bitmap texture applied to it. The bitmap texture will have an alpha channel where A=0 is stuff where the background should show through.

1b. Set up alpha testing as previously described so that any pixels in the bitmap that have A=0 won't be rendered and won't make it through to the stencil test/write or Z test. The bit in bold is the important behaviour which makes the technique work.

1c. Set D3DRS_STENCILFUNC to D3DCMP_ALWAYS so that the stencil comparison for the screen-door pass always succeeds.

1d. Set D3DRS_STENCILREF to 1.

1e. Set D3DRS_STENCILPASS to D3DSTENCILOP_REPLACE so that any pixels that pass the stencil test wrill write the reference value (1) into the stencil buffer. They will all pass, except any which failed the alpha test.

1f. Set D3DRS_STENCILENABLE to TRUE to enable the stencil buffer.

1g. If you don't want the actual pixels of your mask to be visible, use the D3DRS_COLORWRITEENABLE render state to disable the colour from making it through to the frame buffer or enable alpha blending and a ZERO:ONE blend (they will map to the same fast-path thing in most drivers, the latter is better supported on ancient drivers though)

1h. Now the states are set up, render your mask quad - either aligned to the screen or moved with the 3D object, depending on which effect you're trying to achieve.

2a. Set D3DRS_ALPHATESTENABLE to FALSE to disable alpha testing (assuming your 3D object(s) doesn't require it).

2b. Set D3DRS_STENCILFUNC to D3DCMP_EQUAL so that the only pixels which pass the stencil test are those which match the alpha reference value that was written in step 1.

2c. Render your 3D geometry and it will be masked as appropriate.

Share this post

Link to post
Share on other sites
Wow!!! I don't know how to thank you enough for taking the time to explain all this.

3a is definetely what I do NOT want. It's either 3b or 3c. Hadn't thought about the mask moving with the object. On second thought, 3b is probably what I want.

I'm going to have to digest all of this first.

Thanks a million!

Share this post

Link to post
Share on other sites

Clearly, I still haven't got it right. Let me review the steps here and you can tell me where I'm going wrong.

1) The first thing I do is create a device that supports stencil. The related presentation parameters are set to

d3dpp.EnableAutoDepthStencil = TRUE;
d3dpp.AutoDepthStencilFormat = D3DFMT_D24S8;

My understanding is that this will give me 24 bits for the z buffer, and 8 bits for the stencil buffer.

2) Now I set a texture for the stencil. The way I've been doing textures is to use D3DXCreateTextureFromFileInMemory, where, in memory, I create a 32-bit BMP version of my 1-bit bitmap. I set the 3 colors to 0 in all cases, and the alpha value to either 0 or 255.

3) I then use SetTexture (1,mytexture)

I use "1" because I'm reserving "0" for my true textures and I want to keep them separate. Besides, I'm only going to draw one quad and then dispose of the texture.

4) Now comes the part where I'm getting confused. I understand I have to use SetSamplerState, so I can set D3DSAMP_MINFILTER/MAGFILTER to D3DTEXF_POINT, and D3DSAMP_ADDRESSU/ADDRESSV to D3DTADDRESS_WRAP.

Question: what does the first argument to this function stand for? I'm leaving it at zero, but I don't know how this relates to the texture I just defined. If this is a global setting, how do I relate it to multiple textures?

Quetion: How about SetTextureStageState, to set D3DTSS_COLOROP? Do I even need to set it? And the first argument has the same issue as the question above.

5) Question: When do I set and reset all the RenderState flags (e.g., D3DRS_STENCILFUNC, etc.)? I understand I need to first draw my single quad to the stencil buffer; then I draw my regular geometry to the back buffer. So I assume that once I've drawn to the stencil buffer I can get rid of the texture I created and won't need some of the flags anymore. But I will need others when drawing the geometry. Could you clarify this for me?

Right now I simply lost my image, like the stencil buffer, if it's doing something for me, it's just blocking everything.



Share this post

Link to post
Share on other sites
I have figured this out and, for completeness, I'm posting here a pseudo-code that does what's needed. I call it pseudo-code because I extracted it from my code, and I had to simplify a few things to make it more readable.

Here it is. Thanks to all who helped in the right direction!

/* buffer size... get it from somewhere */
int xsize, ysize;

/* bit pattern sizes (usually, 32x32) */
int width, height;
int size, ntype, npts;

/* clear stencil buffer */
device->Clear( 0, NULL, D3DCLEAR_STENCIL,

/* set parameters that will change stencil buffer on non-zero bits */
float pointsize = 1.0f;
device->SetRenderState (D3DRS_COLORWRITEENABLE,0);

/* set point type */
size = 5;
npts = 1000;

/* loop over squares */
n = 0;
jx = jy = 0;
for(ix=0; ix<xsize; ++ix) {
jx = ix % width;
for(iy=0; iy<ysize; ++iy) {
jy = iy % height;
if(bit(jx,jy)) {
if(n == 0) {
bfr[size*n ] = (float)ix;
bfr[size*n+1] = (float)(ysize-iy);
bfr[size*n+2] = 0.f;
bfr[size*n+3] = 1.f;
color = D3DCOLOR_COLORVALUE(1.f,1.f,1.f,1.0f);
/* check for flushing buffer */
if(n == 1000) {
n = 0;
if(n) {

/* reset states */
device->SetRenderState (D3DRS_COLORWRITEENABLE,

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!