I've written a antialiased floating point software renderer that uses an a-buffer (that's an antialiased, area averaged, accumlation buffer, NOT accumulation buffer) type mechanism to do pixel subsampling and I'm having problems where you have triangles of two different materials of a mesh meeting. Sometimes bits of the background shows through on common edges. I suspect this is because of floating point round off errors when determining which of the subsample points belong to which triangle. If you have an edge that's shared between two triangles with different materials due to floating point roundoff errors sometimes a subsample point belongs to both triangles or neither triangle. Does anyone have any hints/experience/references on how to resolve this kind of problem?

From my experience the only way to avoid any kind of numerical problem on a sub pixel level is to use fixed point instead of floating point. Floating point can still be used for all vector math, down to vertex level. But as soon as it goes into edge tables, sub pixel positions and coverages, fixed point is the only way to get rid of unpredictable and unreliable behaviour.