For one, I fixed some more issues with the lightmap stuff.
Here is a shot of a lightmap texel visualization mode, where each point represents a lightmap texel center, and the color indicates if it was a border area, etc.
One of the main problems that I had with the lighting and occlusion map code is that a polygon soup does not have a nice clean idea of inside/outside the world mesh, like a BSP would.
The upshot of this is that it's difficult to find a good bias direction when lighting interior corners of rooms.
For instance, if you simply bias your texel->light raycast along the triangle normal on the corner of a flat wall, you will simply bias into the other wall, not away from the corner.
So, my fix for this was to make a fully vertex-shared mesh for the level for lighting purposes. Each vertex had its normals averaged with all other vertices that shared the position. This has the effect of making corner vertices point out 45 degrees, which is a great bias direction.
All this worked fine to prevent corner seams for lighting.
Once I had this 'adjusted normal', I used it to generate a hemisphere for calculating ambient occlusion. A few weeks ago I made a spherical test that didn't need a specific normal direction, but this was well more than twice as slow at the same # of raycasts, and needed 8x the # of raycasts to look good as well.
A problem with the hemisphere approach is that paradoxically, interior corners were brighter than the walls next to them, whereas they should be dimmer. This was caused by using too many rays in the normal direction, as well as the 45 degree rotated hemisphere. The adjusted hemisphere could 'see' more of the level than the walls next to it.
Yesterday I realized the simple fix : use the adjusted normal direction only for pushing raycasts away from the surface, but use the original flat surface normal for the occlusion hemisphere direction.
There are still some issues that would be better solved by having a definite inside-outside test, which would allow a clean bias normal even in the presence of non-manifold geometry. I am considering several approaches, one being adding tools to the editor or engine to clip tris to each other to avoid mesh->triangle boundary issues.
Also props to JollyJeffers who suggested using gutters for lightmaps. It does, as I replied cause shadowing artifacts, but it also is faster to compute & reduces occlusion artifacts, which are more obvious, so I have changed to copying valid texels into invalid regions via a dilation filter instead of lighting past the edge of a lightmap triangle.