The entity lighting system was designed primarily for characters, which are relatively small, so I did the character shadowing through cpu raycasts to dim the light color as the character becaome more occluded. This is still fine for characters, but not for large doors or moving bridges.
Another minor issue with larger entities is their lack of decals. If you can blast a hole into a stone wall, you should be able to make an impression on a wooden door, but since they were rendered the old way, and weren't even bump mapped, making decals work on them was impractical.
A couple of days ago, I took the first step in converting the code. I made a back up. The next step was to calculate tangent space for the meshes, and convert their vertex format to the same format as the world geometry.
Once that was working, I refactored the entity drawing code to be more modular, and, since the entities would be drawn more often if they cast shadows on each other, I converted the geometry to a static vb, rather than copying it into a dynamicvb each time it was drawn.
Next, I converted the entity lighting to use a similar path to the world geometry, so the entities now draw their diffuse colors & normals during the compositing step, so they could potentially have decals applied.
One of the things that absolutely sux about depth-based shadows is self-shadowing artifacts. In my case, I could try to minimize them by rendering only back faces to the shadow map, but that might not have solved the self-shadowing problem with world geometry.
Here's how the system works :
For each shadow casting entity ( door, char, etc. )
Create a frustum from the light to this dude's bounding sphere, out to the light range
Use this frustum to gather world geometry he might cast onto
If he receives shadows from the world,
Use this frustum to gather world geometry he might receive from
If he receives shadows from other entities,
Use this frustum to gather entities he might receive from
Now, we don't necessarily want the casting geometry to self-shadow. In the case of the door, it's redundant, and in the case of a character, it makes him harder to see, plus the inevitable self-shadow artifacts.
And with world geometry, a world chunk may be in both the casting and the receiving lists, so there needed to be a way to separate casters from receivers to avoid self-shadowing problems on the world.
One approach would have been to do separate renders for casters & receivers, but that would be slower than the solution I ended up using - render the receivers & the casting entity to alpha, and render the casting world geometry & entities to blue. That way the same render target is used, and just different shadow testing shaders can be used. This also allows one to do self-shadowing or not on the caster based on which channel is
Here is a shot of the door shadow caster render target. See how some of the other objects in the scene are rendered to the blue channel to cast shadows onto the door. The door is rendered in red ( and alpha ) so the world geometry it can test against the caster only.
And here is a shot of the door receiving entity shadows, and the world shadow ( the lower left corner of the door ).
In the process of all these changes, I introduced a few bugs, some of which are still to be addressed. Here's one where the fog shader didn't reset the world matrix to identity :