I thought it might be interesting to go into some details on the trials and tribulations of shadowing in my engine.
I believe the approach to shadows can have the most far-reaching impact on the rendering engine - beyond any other particular rendering feature.
Through the development of the engine that became the basis for Ancient Galaxies, I've tried many different approaches before settling on the current method.
The first main approach I took was per-vertex shadowing. I didn't care about moving lights, so I thought if I could burn the shadowing terms into the vertex data, I could have multiple shadowed lights cheaply.
My approach was to store up to 7 scalar terms in each vertex, stuffed in diffuse.rgba and specular.rgb. Each term stored an occlusion term from [0..1] with respect to a certain light. Obviously this restricted the engine to only handle shadows from 7 lights at a time. Because my levels were broken up into roughly 8x8x8 meter chunks, this was not a limitation in practice.
The idea was to perform 7 different lighting calculations per-vertex in the vertex shader, and scale each one by the appropriate occlusion term, then sum them all up to get the total lighting for that vertex.
This worked very well, and looked good, with a couple of caveats. First of all, my world was a bit over-tessellated at the time, down to 32 triangles per meter, even in flat areas. This was to keep the lighting looking good. Later, I changed the attenuation function to be more linear, and I was able to reduce the tessellation quite a bit, to 8 triangles per meter.
This was a good solution for semi-static shadows. Lights could turn on & off, or change color or intensity, at no extra run-time penalty.
When I later played around with different sunlight directions on some test levels, I discovered that the lighting only looked good b/c I happened to be testing a 45 degree light angle that perfectly matched the tessellation of my level. If I rotated it 90 degrees, it looked like a blocky mess. Even going back to the highly tessellated version didn't help enough.
Here is a shot from a version of the engine with per-vertex shadows, and a less-than ideal sunlight direction.
Plus, I had yet to put trees or more organic shapes in the level, so I realized I was having problems with the best-case scenario. I really wanted trees, etc, and although I was confident they would be able to receive shadows fine, unless the floors and walls were over-tessellated or specially clipped to the shadow boundaries, this wouldn't be super-practical.
Now, it might seem funny that I'm worried about how many triangles in a small overhead level, but part of the issue is collision detection. I really didn't want multiple versions of the level geometry - one for rendering, and another for collision. I have yet to find an easily integrated algorithm for simplifying geometry that does not at some point break planarity of the underlying mesh. For instance, the D3DX progressive mesh stuff certainly starts leaving out important border parts of a flat floor before it goes after some over-tessellated walls. Perhaps these things are buggy, or perhaps they are just designed for character-type meshes and not world geometry.
One of the neat things my engine does is proper collision - not only for characters, but for sparks, smoke, debris, etc. Having overly tessellated geometry limits how much of this I can do. If someone reading this has a pointer to a robust algorithm for simplifying, I'm all ears. I have written one myself, but it is extremely slow, so I'm not currently using it in the art pipeline.
I didn't want the level creators to have to keep two versions of the geometry in sync, so it was either automate the simplifcation, or keep triangle counts in check.
So, it seemd per-vertex shadows weren't going to work at all. Tomorrow I'll update with the fun I had with doing shadows light-map style. Later in the week I'll talk about the character shadows.