i have a code (that doesnt work for a big scenarios) and i think it has something to do with the culling problem because i may wrongly set the clockwise order, the problem now is how do i force clockwise as front for a side caps of shadow volume
You can issue before a draw call in opengl what winding is culled, and which is rasterized. You can cull CCW, or CW, or none, or, both and render nothing.
Generaly a shadow volume of z-fail technique is performed on visible rendered frame in a way of getting occluder rendered twice, but altering only stencil buffer, not color and depth one. It should be done like this:
clear stencil buffer per every light with some reference value (for our example 127)
(for first extruded geometry pass )
-set culling to cull CW (render front faces of yours only)
- set depth test to pass when pixel depth is greater (gets behind depth buffer value)
- decrease stencil value by one
(for second extruded geometry pass )
-set culling to cull CCW instead (render back faces of yours only)
- set depth test to pass when pixel depth is greater (gets behind depth buffer value)
- increase stencil value by one
now everywhere where stencil buffer value is larger than 127, is a shadow for the light. You can continue render other occluders of the light to the stencil without worrying for this to be not true (>127 shadow, <=127 lit).
Winding of faces is something that stays consistent across geometry and that is all that really matters. But for correct artefact-less stencil shadows, you will need yet all your casting geometry to be an enclosed mesh, what is not a big problem, but what is worse, you will need to index additional two triangles per every edge, while having all position verticies unique (vertex count will equal 3*trianglecount), else you will not extrude geometry volume properly.