Sign in to follow this  
synth_cat

stencil shadow issues w/ textured quads

Recommended Posts

Hello all, I am currently trying to add "real" shadows to my game CellZenith. The shadow casting entities in this game are all flying ships, each of which is composed of several quads on top of each other. Originally, each ship's shadow was basically just a dark smudge (single alpha-blended quad.) Now, however, I am trying to change this so that every single quad in each ship casts a perfect shadow of itself. When I tried to do this, I got a problem: the shadows for the ship's quads became darker where they overlap (like pieces of tissue paper.) This obviously calls for a stencil buffer, but I don't know exactly how I should set that up. I currently have it going like this:
		d3ddev->SetRenderState(D3DRS_STENCILFUNC,D3DCMP_EQUAL);
		d3ddev->SetRenderState(D3DRS_STENCILREF,0x0);
		d3ddev->SetRenderState(D3DRS_STENCILMASK,0);
		d3ddev->SetRenderState(D3DRS_STENCILPASS,D3DSTENCILOP_INCR);
		d3ddev->SetRenderState(D3DRS_STENCILFAIL,D3DSTENCILOP_KEEP);
		d3ddev->SetRenderState(D3DRS_STENCILZFAIL,D3DSTENCILOP_KEEP);


This doesn't exactly work, though. Basically, if one shadow quad is drawn before a smaller one in the same area, the smaller one will be completely invisible (because the one before it has already INCR'd the stencil buffer over that entire area.) What is the correct way to handle the stencil buffer in this situation? I'm sorry for the lack of images - if I knew a site where I could easily post pics I would have done. Thanks, Greg

Share this post


Link to post
Share on other sites
Frankly I think this is the wrong way to approach this. I dont see the stencil buffer as a good idea.

Instead why not use a kind of simple shadow mapping... render the shadow casters from the light's position and project it on the whole scene. Since you are already rendering the sahdow sprites, the costs here are going to be fairly low.

Share this post


Link to post
Share on other sites
I am not an expert nor in shadowing nor in Stencil buffers, but I do believe that setting the STENCILMASK to 0 is useless because then the whole comparing will not work.
The compare function compares 2 values, one of them is the current stencil buffer value at that point, ANDed with the STENCILMASK.
So, if the STENCILMASK is 0, the value of the stencil buffer at that point is ignored, and the result will always be 0.

Share this post


Link to post
Share on other sites
Quote:

I am not an expert nor in shadowing nor in Stencil buffers, but I do believe that setting the STENCILMASK to 0 is useless because then the whole comparing will not work.
The compare function compares 2 values, one of them is the current stencil buffer value at that point, ANDed with the STENCILMASK.
So, if the STENCILMASK is 0, the value of the stencil buffer at that point is ignored, and the result will always be 0.



This is the comparison made:

(StencilRef & StencilMask) CompFunc (StencilBufferValue & StencilMask)

Here, CompFunc is ==.

StencilRef and StencilMask are both 0, so I'm assuming (StencilRef & StencilMask) always returns TRUE. The second part, (StencilBufferValue & StencilMask) will only return TRUE for pixels which have not been written to yet (because StencilBufferValue will be set to 1 every time a pixel is drawn.)

Quote:

Instead why not use a kind of simple shadow mapping... render the shadow casters from the light's position and project it on the whole scene. Since you are already rendering the sahdow sprites, the costs here are going to be fairly low.



I'm not sure what you mean - there are no actual lights.

Share this post


Link to post
Share on other sites
There don't need to be actual lights.. just use the "sun" position, or wherever you want the shadows cast from. You can make a fake sun position if needed...

the point is its easier to make a shadow map this way to mess with the stencil buffer.

Share this post


Link to post
Share on other sites
Quote:

StencilRef and StencilMask are both 0, so I'm assuming (StencilRef & StencilMask) always returns TRUE. The second part, (StencilBufferValue & StencilMask) will only return TRUE for pixels which have not been written to yet (because StencilBufferValue will be set to 1 every time a pixel is drawn.)


**************** 0 & 0 = 0 (FALSE!!!) ****************************

When you AND two numbers, it's enough that just one of them is 0, the result be 0 (FALSE) too.

Share this post


Link to post
Share on other sites
The stencil mask is not really applicable in your case. Just leave it as 0xFF. Don't set it to 0. The operation will always fail. It's used more for storing multiple stencil values per pixel.

You should setup the stencil function to ALWAYS, set the reference to 1, and set the operation to EQUAL. When you draw the quads, the stencil in those areas will always be set to 1. You can also do it the way you are now by incrementing the stencil value and then draw your shadows in areas where stencil is not 0.

I'm just curious. How are you planning to get shadows of the alpha portions of the quads using this method? I haven't really thought about how that could be done. Using the alpha test in conjunction?

Share this post


Link to post
Share on other sites
Quote:
I'm just curious. How are you planning to get shadows of the alpha portions of the quads using this method? I haven't really thought about how that could be done. Using the alpha test in conjunction?


Yes, I am using alpha testing to prevent transparent pixels from affecting the the stencil buffer. At first this didn't work exactly right because semi-transparent pixels were writing to the stencil buffer, which was causing some ugly problems. However, I just increased ALPHAREF and this problem went away. The shadows look great now. I'm glad I didn't have to do shadow mapping because I don't think this rental computer can run shaders . . .

Thanks for all the help, guys!

If you want to see a picture, go here. Devimg.net doesn't let you directly link to images, unfortunately.

[Edited by - synth_cat on August 6, 2007 6:34:01 PM]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this