I've been looking at various forms of shadowing for my game, but I was wondering which ones are personal favorites among developers. I'm trying to make everything in shaders with as little (or preferably no) fixed function rendering. It sounded like I could do stencil shadow volumes using HLSL vertex shaders, but I was unable to find a good source on how to actually implement it. Any pointers, opinions, tutorials, etc?

Shadow volumes is definately the way forward, since it can be so powerfully employed with shaders. Unfortunately, the technology will only really reach maturity with DX10.

The SDK tutorial on shadow volumes is quite good, but the technique employed is a bit of a mind bender. It involves processing the mesh at startup and, using the adjacency information, generating a pair of degenerate polys along EVERY adjoining edge.

These degenerate polys then become the sides of the shadow volume when extruded by the shader.

Shadow volumes are also detailed in the book 'Real Time Rendering Tricks and Techniques in DirectX' by Kelly Dempski, however, they cheekily use a torus mesh to avoid the requirement of the degenerate triangles.

Looking at the DirectX10 docs, the introduction of geometry shaders will do away with all the degenerate polys nonsense, since these can be produced on the fly.

I've decided to use fake shadows until then. :)

The shader is actually quite simple. Get the vector from the light to the object, then for each vert, do dot(lightVec,Normal) to determine if it is facing the light or not. If not, offset the position by some (large) factor multiplied by the light>object vector.

You also need to do several passes of any shadow casters. I didn't include my terrain simply because it became too costly in fps! This means the terrain can't shadow itself, boo hoo.

First, render the object as normal.

Second, render front facing polys (relative to the camera now) to the stencil buffer. If the stencil test fails, add 1 to the stencil buffer.

Third, render all rear facing polys, if the Stencil test fails, subtract 1 from the buffer.

Draw all other shadow casters in the same way, then finally draw a full screen aplha blended grey quad with the stencil buffer. This should only draw where the shadow remains.

Clear as mud?

Good Luck

Simon

EDIT: I amy have the pass/fail, add/subtract bit the wrong way round... from memory!

I've read a couple of papers over at DevMaster.net about stencil shadows, but still am unsure of how to implement it. I know there is a stencil shadow project in the DirectX SDK, but just reading code doesn't work well for me. Does anyone know of a tutorial (in similar fashion to NeHe) that would show me some C++ and HLSL code for doing stencil shadows with a decent explanation of what's going on? If not, I guess I'll try and learn from the DXSDK project.

I'm not sure if you'd want to spend money on a book or not; however, the ShaderX books are very nice for learning shaders and their techniques. In ShaderX2: Tips & Tricks with DirectX9, there is an entire section dedicated to shadows.

Check out Half.com if you don't want to pay full price for the book.