Jump to content
  • Advertisement
Sign in to follow this  

Stencil Shadows Problem

This topic is 888 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello There! i hope someone can help me


i have been trying to get stencil shadows working for months (i know the technique has been largely superseded by shadow maps - but i am set on doing things this way)

i got the shadow volume constructed, i detect the edges that are silhouettes (one light front side and one dark backside)
and then i "extrude" (really, make quads or sets of triangles) that go along the vector that the light travels.

so far so good. in fact, my stencil shadows work perfectly with convex shapes like a box or a circle...

but when i load up something a little bit more complicated like a "T" shape mesh i find that some of the extruded quads go inside other quads in the shadow volume - and this results in my stencil shadows having holes where the shape does not.

please help!!!

at first i thought that the faces didn't have the correct orientation... but they do. i rendered the shadow volume itself as a primitive and i found out the faces do point outwards.

the problem might be the stencil operations.
nothing that i do with the stencil gives me the result i look for.

the stencil operation that i am using now is to increase, render my shape, then flip the culling and increase the stencil again, - and then i render the scene dark, keeping all the stencil values marked as one. so it only draws dark pixels where it was increased the first time. - again - this works great when the shape is something like a box or a sphere (convex) but a 90 degree angles shape has quads coming out of some corners that are inside of the shadow volume, and they mess up my stencil!!!



this is the shape i am using, and i also illustrated the problem with 3d software:






you can see the problem here. - one of the quads i extruded goes inside the shadow volume and makes a weird corner there.

as a result the stencil operation doesn't work well.




here's how it looks in my program.




here i outlined the quad that is giving me trouble:




i have tried everything i can think of. nothing works. how do i make this face not matter?



or maybe the face is flipped after all... i am not convinced of anything to be honest. trying to double and triple check the orientation of the face got me nowhere.




i made one more picture illustrating the issue, the orange edge in this image makes a quad that is undesirable.



Edited by Basteagui

Share this post

Link to post
Share on other sites

I'm no expert on this but I did shadow volumes recently and didn't have an issue with shadow volumes going through themselves. I was shadowing a Menger Sponge fractal. I used the following tutorial, perhaps there is something in it that might help:



The major problem I hit was making sure the quads were facing the correct way. You said you checked this but did you check it on the actual mesh that is giving you problems? I assume you rendered back faces one color and front faces another to see? Do you have front/end caps? I also found I needed to use a 'sealed' mesh, is there a split around that quad that is causing you problems?

Edited by Nanoha

Share this post

Link to post
Share on other sites

What Eric said. From a tutorial i was reading a long time ago:


Obviously we can't cast a ray for every object in OpenGL, but we can get the same effect by using the stencil buffer. We must render the shadow volumes in two passes, one for the front facing polygons and one for the back facing polygons. Every front facing poly is where the ray would intersect a shadow volume, increase the stencil buffer here. Every back facing poly is where the ray would exit the shadow volume, decrease the stencil buffer here. After rendering both passes, any pixel where the stencil count is 0 is NOT in shadow.


Also, it helps to add some debug normals to your volume polys, to make sure that they are facing the right way.

Edited by mathematical

Share this post

Link to post
Share on other sites

i appreciate the support guys. i didn't respond earlier because this was making me feel really dumb... and frustrated, it was hard to even keep working on this.




however it wasn't flipped polygons or any of that.


i believe the main issue i had was that you need to enable depthbuffer BUT DISABLE WRITETODEPTHBUFFER...


this is something that i didn't find in any of the tutorials... someone in gamedev chat mentioned this to me and i didn't know what he meant until after months of trying different stuff and following different leads.



i still have a problem though my stencils are flipped... where i should have dark areas i have light areas and viceversa!!


maybe someone could help me finally solve this once and for all and flip the stencils?? everything i try just breaks it even more.


i tried to clear the stencils with all 1s instead of 0s but it doesn't even make a difference at all... that makes no sense!!!!


i am coding with xna

i really need help with this final detail though... if i could flip this thing it would be perfect!



i got a 500 line code and a small fbx model made of cubes. but i am not sure if i am allowed to post code here or upload stuff...





this is what it looks like... where i should have gray i got black and viceversa!

Edited by Basteagui

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!