DISCLAIMER: I'm using Unity, but I just need the theory here, really. As a warning on the Unity player, there's no invisible walls, so please don't fly off the edge of the world.

Unity player link

I've been trying to use precomputed stencil volumes like spheres, boxes, and cylinders to render holes in meshes, but I'm having a bit of trouble. The big problem is that I've actually never coded stencil shaders before and usually use visual shader tools like the UE4/UDK material editor and Shader Forge, but those aren't an option for this project.

I'd like to accomplish something like in the Unity player there where I'm cutting a spherical (or other geometric) volume out of the level geometry. It seems to me that I should be able to use something like Carmack's reverse for this and just have it so that instead of calculating the shadowed areas, I'm calculating an alpha clip on the intersected geometry. Because I'm on Unity Free, I don't have access to things like render textures, but I do have access to the stencil buffer, which should make this theoretically possible.

I think.

...help

Are you aiming for CSG, perhaps something like the following?

ftp://ftp.sgi.com/opengl/contrib/blythe/advanced99/notes/node22.html

Are you aiming for CSG, perhaps something like the following?

ftp://ftp.sgi.com/opengl/contrib/blythe/advanced99/notes/node22.html

I had that open a while ago, but I think it's a bit more than what I need. All I really want to do is discard intersected pixels, while this goes so far as to compute all new faces. Plus I'm only handling subtraction and not an entire CSG set, which is something I'm already writing in the background for collision handling.

You should definitely be able to do that.. by first drawing all your visible geometry only to the depth-buffer, then drawing the subtracted object like a stencil volume, and then finally drawing all your visible geometry again to the color-buffer. Set stencil-testing for the visible pass to only allow geometry outside of the shadow volume (same as when using stencil-shadows with additive lighting passes).

You should definitely be able to do that.. by first drawing all your visible geometry only to the depth-buffer, then drawing the subtracted object like a stencil volume, and then finally drawing all your visible geometry again to the color-buffer. Set stencil-testing for the visible pass to only allow geometry outside of the shadow volume (same as when using stencil-shadows with additive lighting passes).

Okay, I was hoping that was the case. Now for the tricky bit. I plan on having the players be able to lay down multiple holes that may intersect. The main reason I'm looking at Carmack's reverse over other solutions is because it doesn't seem to suffer from the same artefacting issues as depth-pass and exclusive-or. I trust I won't have any severe issues if I have multiple overlapping holes at different depths?

I have implemented stencil shadows by carmacks reverse since z pass tecnique cannot render shadow if observer is in the volume of the shadow. I do not compute the geometry of the volume, I only move verticies on gpu in light direction if they are on trinagles facing away from light. This of course has a demand on mesh defintion, it has to be an enclosed mesh (open meshes are highly metaphysical, but allows for certain optimization) with fulll triangle indexing and even reindexing additinial edge trinagles. It is then a trivial operation in vertex function but what makes shadow volumes computationly difficult is fill rate in pixel function. Since you do not compute anything (adding two vectors in vertex function), and in pixel function you output only a single value- this particular operation of writing into a memory is heavy demand.