So I have reimplemented my old shadow volumes algorithm--mainly for completeness. This implementation uses a geometry shader to generate depth-fail (read: two-ends capped) geometry.
All works well, except that for some glancing angles, the geometry is extruded out from where it shouldn't. This results in artifacts. This seems to happen particularly where a triangle's edge that should be shadowed actually thinks it's a silhouette edge.
Here's a simple example. All seems well:
But rotate it, and an artifact develops:
Here's a closeup of such an artifact:
The dark face at the bottom should be completely shadowed, and therefore should not generate an edge along the bottom-left. It generates one anyway, screwing up the render.
The normal used to compare to the light direction is generated from the cross product of the geometry itself, so it's not an interpolation issue. Also, the artifacts seem to only happen in a narrow range of glancing angles. The same face will correct itself if turned further.
My question is, is this a common thing? The only thing I could think of is a precision issue, but the error is really quite large. I realize I haven't given much information about the implementation, but does anyone seen this sort of thing with volume extrusion before?
-G
