I'm using the stencil buffer to draw the intersection of a plane and a sphere. However I'm getting some annoying artifacts near
the boundary of the intersection. See the attached image.
These artifacts depend on the camera position. And although they look somewhat polygonal they generally don't coincide with
the edges of either mesh.
Here's the algorithm I'm using:
Depth test is enabled, culling is disabled.
- Draw the plane mesh with stencil.func.ref = 1 and stencil.func = GL_REPLACE.
(ie. fill the stencil buffer with 1)
- Draw the sphere with depth.mask = false and
stencil.op.zpass = GL_INCR for front faces and stencil.op.zpass = GL_DECR_WRAP for backfaces
(I use glStencipOpSeparate and draw the sphere once)
- Blend fullscreen quad where stencil value is greater than 1