I fixed a bunch of bugs over the last couple of days. Things are really shaping up. I had to fog the entities again, after
changing their lighting path. I also fixed some issues with the water, including making sure the fog was working properly.
A few more editor fixes as well.
The main fix was to the entity lighting & shadowing system. There were a couple of bugs that were driving me insane.
One bug I fixed yesterday was with the z range of my shadow maps. I seemed to be getting z fighting, and I thought that maybe 8-bit wasn't enough. When I really looked at it, though and ran some tests, that wasn't the problem. Turns out the z buffer attached to the shadow map rendertarget had a poor z range that was throwing away precision. I made differing settings for point lights vs. directional lights as well.
Sometimes, and not in any one obvious scenario, a light shadow would leak through the back of a door. Like if a door were closed, and the camera were facing the door, with the player hidden behind the door, the player would seem to project a shadow on the wrong side of the door.
At first I thought it might be a tangent space issue, since if I inverted the light direction, it seemed to fix it on one level.
This was ironic, as I was one of the first people to do this sort of tangent space lighting back in 2000, and now my implementation seemed buggy.
But, I output the normal, tangent, and binormal as colors, as well as the lights in various spaces, and it seemed to be working fine.
Turns out the problem was with the occlusion term. The lighting system for the world geometry works like so :
1) For each light, an 8-bit occlusion * attenuation term is calculated for the lightmap for one world chunk.
2) Once all world geometry has these terms calculated, for each world geometry triangle, make sure at least one texel in the triangle's occlusion map actually contains some light.
3) Make a new index list, containing only triangles that are partially lit with respect to this light. This way you save fill rate & geometry rate when you render lighting & shadows. Also, make a bounding box of the surviving triangles to cull better.
4) When rendering the ambient & emissive pass to the frame buffer, also stick the sunlight occlusion term in dest alpha
5) For all lights after the main sunlight, do a separate pass to render occlusion * attenuation term into dest alpha
7) Then create shadow maps, and write a zero or one to dest alpha, modulating with what is already there. This is how dynamic shadows are added to the scene
8) Then perform lighting, using dest alpha to control where light is added
9) Go to step 5 and repeat until all lights are done
For the entities, I added a step to pass #5 to render their attenuation term to dest alpha. I also had to add the self-shadowing term, to prevent specular from coming through the side of an object facing away from the light.
The main bug was that I was not rendering the entities' attenuation * self_shadowing term to dest alpha for the main sunlight. Once I fixed that, things finally were working right.
Here are some shots of the working doors with no extraneous shadows :