Question regarding Stencil Buffer in DirectX 9

Started by
4 comments, last by Nik02 11 years, 9 months ago
I heard I can specify the stencil mask using a black and white image file. How do I do that and how do I specify the area of maksing?
Advertisement
Enable the stencil write render state, render your mask texture and discard those pixels that you don't want to write in the stencil buffer by using the clip() pixel shader function. The pixel shader logic specifies the "area of masking".

In your case, you'd sample the mask texture, and feed its luminance - 0.5f (where 0.5 is the gray treshold) to the clip function as a parameter. The clip function discards the currently processed pixel if its parameter is less than zero, and effectively prevents it from being written to color targets, depth buffer or the stencil buffer.

Alternatively (in case you don't want to use pixel shader), enable alpha testing and set the alpha test treshold to 0.5. Then output the texture luminance (or alpha, if you have an actual alpha channel in the texture) as alpha in the texture stage cascade. Alpha test rejection effectively establishes the same thing as the clip() function.

Niko Suni

I am not using lighting... I need the stencil buffer for minimap.
Neither of the techniques I described are in any way related to lighting.

In order to write something to the stencil buffer (without card-specific hacks), you need to render something, because the standard stencil buffer formats are not lockable and therefore you can't get direct access to their surface memory from within your program. Rendering can be thought as an indirect write access.

Niko Suni

Sorry my bad. When I saw the word Luminance, I thought there's some relation in lighting. Now this sentence make me even more confuse.

"Rendering can be thought as an indirect write access."
Rendering something is the only standard way to write an image (such as a mask) to a stencil buffer, because you can't access the stencil buffer memory by locking it.

Some hardware and drivers do allow locking of stencil buffers when they are allocated in a special way, but in D3D9 this is always done via vendor specific extensions (thus forcing you to write custom code for each vendor such as AMD and NVidia). In contrast, the technique(s) I outlined only require that the target GPU supports pixel shaders or alpha testing (in addition to obviously supporting stencil buffers). Any GPU less than about a twelve years old can do that, in a standard way.

When I said "luminance", I meant the white level of the texture you want to use as the mask image. If you clip by (luminance - 0.5f), all pixels that have less than 50% white on the mask image will not be rendered to either the color targets or the depth/stencil buffer. The clip simply provides the means to control which pixels get written to the stencil buffer - after you've filled it, you can use it to mask subsequent drawing operations.

Note that you could also use an alpha mask texture directly instead of using a stencil buffer.

Niko Suni

This topic is closed to new replies.

Advertisement