Hello There! i hope someone can help me

i have been trying to get stencil shadows working for months (i know the technique has been largely superseded by shadow maps - but i am set on doing things this way)

i got the shadow volume constructed, i detect the edges that are silhouettes (one light front side and one dark backside)
and then i "extrude" (really, make quads or sets of triangles) that go along the vector that the light travels.

so far so good. in fact, my stencil shadows work perfectly with convex shapes like a box or a circle...

but when i load up something a little bit more complicated like a "T" shape mesh i find that some of the extruded quads go inside other quads in the shadow volume - and this results in my stencil shadows having holes where the shape does not.

please help!!!

at first i thought that the faces didn't have the correct orientation... but they do. i rendered the shadow volume itself as a primitive and i found out the faces do point outwards.

the problem might be the stencil operations.
nothing that i do with the stencil gives me the result i look for.

the stencil operation that i am using now is to increase, render my shape, then flip the culling and increase the stencil again, - and then i render the scene dark, keeping all the stencil values marked as one. so it only draws dark pixels where it was increased the first time. - again - this works great when the shape is something like a box or a sphere (convex) but a 90 degree angles shape has quads coming out of some corners that are inside of the shadow volume, and they mess up my stencil!!!

this is the shape i am using, and i also illustrated the problem with 3d software:

you can see the problem here. - one of the quads i extruded goes inside the shadow volume and makes a weird corner there.

as a result the stencil operation doesn't work well.

here's how it looks in my program.

here i outlined the quad that is giving me trouble:

i have tried everything i can think of. nothing works. how do i make this face not matter?

or maybe the face is flipped after all... i am not convinced of anything to be honest. trying to double and triple check the orientation of the face got me nowhere.

i made one more picture illustrating the issue, the orange edge in this image makes a quad that is undesirable.

What Eric said. From a tutorial i was reading a long time ago:

Obviously we can't cast a ray for every object in OpenGL, but we can get the same effect by using the stencil buffer. We must render the shadow volumes in two passes, one for the front facing polygons and one for the back facing polygons. Every front facing poly is where the ray would intersect a shadow volume, increase the stencil buffer here. Every back facing poly is where the ray would exit the shadow volume, decrease the stencil buffer here. After rendering both passes, any pixel where the stencil count is 0 is NOT in shadow.

Also, it helps to add some debug normals to your volume polys, to make sure that they are facing the right way.

i appreciate the support guys. i didn't respond earlier because this was making me feel really dumb... and frustrated, it was hard to even keep working on this.

however it wasn't flipped polygons or any of that.

i believe the main issue i had was that you need to enable depthbuffer BUT DISABLE WRITETODEPTHBUFFER...

this is something that i didn't find in any of the tutorials... someone in gamedev chat mentioned this to me and i didn't know what he meant until after months of trying different stuff and following different leads.

i still have a problem though my stencils are flipped... where i should have dark areas i have light areas and viceversa!!

maybe someone could help me finally solve this once and for all and flip the stencils?? everything i try just breaks it even more.

i tried to clear the stencils with all 1s instead of 0s but it doesn't even make a difference at all... that makes no sense!!!!

i am coding with xna

i really need help with this final detail though... if i could flip this thing it would be perfect!

i got a 500 line code and a small fbx model made of cubes. but i am not sure if i am allowed to post code here or upload stuff...

this is what it looks like... where i should have gray i got black and viceversa!

hope it's not disallowed but i got my code on pastebin here:

http://pastebin.com/q2mVNy8H

and i uploaded my fbx to sketchfab which is similar to sketchup warehouse i think -

https://skfb.ly/QxMB