hello, i am using an FBO to where i to render my shadow map to. since the viewport of the light-camera is limited, objects in the far distance dont have a shadow. now to prevent these hard edges where the shadowmap just stops, i want to blur it, so that the shadow fades out at the end. how can i achieve that effect? thanks!

You could use Variance Shadow Maps to get the soft edges. Basically, it involves storing the depth and depth squared in two components of a floating-point texture, in place of the standard one-component depth texture. Then when applying the shadow map to your geometry during lighting, you use a probability equation, described in the paper and demo, to calculate the amount of shadow, instead of the standard yes/no depth condition. You need to use shaders for this technique.

Alternately, you can use a screen-space blur to make them soft. Basically, you render the shadow map applied to the geometry off-screen and blur it with a shader before applying it to the framebuffer. This technique suffers from halo effects where shadowing at object edges gets blurred with the background shadowing.

Edit: I think I may have misunderstood your question. You basically want to fake a penumbra effect where the shadow gets lighter as it gets further behind the occluder?

If you're using a shader to compare the depth, you could subtract the fragment depth from the sample depth to give you the difference in depths. Then mix that with the result from your regular shadowing equation, such that as the difference increases, the amount of shadowing becomes less. This would of course only take into account the closest occluder to the light source, and it would ignore the size of the occluder, so its probably of limited use.

[Edited by - dcosborn on November 25, 2007 3:46:39 PM]

Yes, see the edit in my post for that, although my solution doesn't really handle the edges specifically, just fading out over distance. For the edges of the shadow map, you could use the texture coordinate of the shadow map to create a fade value at the edges and multiply it with your result from the regular shadow comparison.

The following code seems to work for me.


Multiply FadeShadowBoundary(...) by the result of your regular depth comparison.

[Edited by - dcosborn on November 26, 2007 2:30:53 PM]