Finally got around to add an ambient occlusion baker to the pipeline.
I'm baking it all to vertex maps, as these give generally good results and avoid the hassle and memory demands of having to assign unique texture space to every triangle in a mesh. Meaning I could no longer use the render-to-texture-only routines provided by Modo. Also, lots of instanced meshes (grass, trees, etc.) don't exist until mesh compilation time, which caused them to be excluded from any in-modeller performed ray tests. And then I don't like having to bake all that gpu stuff by hand, so this was always going to be automated one day.
Occlusion testing itself is pretty straightforward, but here are some findings specific to creating per-vertex maps:
- Don't only evaluate rays at vertex positions, but sample from various points over the surface of the vertex' neighboring triangles - this avoids local undersampling of the surrounding geometry. Worse case example: most of a vertex' connecting face area may actually be un-occluded, but if the vertex itself is occluded by some small object, all rays fired from its position will be too and the vertex will show up completely dark. Multisampling smooths out such deficiencies.
- Use a series of sample positions combined with a series of ray directions that spread out evenly over respectively a triangle's area and a halfsphere - precomputed Poisson distributions work well. It's not too relevant which point fires which ray; triangles tend to be small compared to their environment, which keeps rays from converging too much once they fan out into the scene.
- Once you have a bunch of per-triangel occlusion values, you'll need to weld them together to form per-vertex values. Normals are usually welded based on some angle threshold and my first idea was to use a similar threshold, but specific to occlusion values. Turns out it's better to have them weld along with the normal angles: if the normals weld, the occlusion values weld too (or any weight/color map value really). What this really means is that you use any lighting discontinuities caused by unwelded triangle normals as an excuse to keep more triangle-specific occlusion data around as well (not averaging at the vertex will result in higher graphical fidelity), but only then, as to avoid a too-faceted look. It also results in the least amount of duplicated vertices needed in a vertex buffer.