Hello There! i hope someone can help me
i have been trying to get stencil shadows working for months (i know the technique has been largely superseded by shadow maps - but i am set on doing things this way)
i got the shadow volume constructed, i detect the edges that are silhouettes (one light front side and one dark backside)
and then i "extrude" (really, make quads or sets of triangles) that go along the vector that the light travels.
so far so good. in fact, my stencil shadows work perfectly with convex shapes like a box or a circle...
but when i load up something a little bit more complicated like a "T" shape mesh i find that some of the extruded quads go inside other quads in the shadow volume - and this results in my stencil shadows having holes where the shape does not.
at first i thought that the faces didn't have the correct orientation... but they do. i rendered the shadow volume itself as a primitive and i found out the faces do point outwards.
the problem might be the stencil operations.
nothing that i do with the stencil gives me the result i look for.
the stencil operation that i am using now is to increase, render my shape, then flip the culling and increase the stencil again, - and then i render the scene dark, keeping all the stencil values marked as one. so it only draws dark pixels where it was increased the first time. - again - this works great when the shape is something like a box or a sphere (convex) but a 90 degree angles shape has quads coming out of some corners that are inside of the shadow volume, and they mess up my stencil!!!
this is the shape i am using, and i also illustrated the problem with 3d software:
you can see the problem here. - one of the quads i extruded goes inside the shadow volume and makes a weird corner there.
as a result the stencil operation doesn't work well.
here's how it looks in my program.
here i outlined the quad that is giving me trouble:
i have tried everything i can think of. nothing works. how do i make this face not matter?
or maybe the face is flipped after all... i am not convinced of anything to be honest. trying to double and triple check the orientation of the face got me nowhere.
i made one more picture illustrating the issue, the orange edge in this image makes a quad that is undesirable.