I've been wrestling for some time on how to produce decent quality, self-shadowing terrain that effectively shadows all other objects in the world. A couple of days ago I had an idea.
For any given point on the terrain, the direct sunlight contribution can be calculated thus:
While the sun is below the horizon or behind occluders (hill, mountains etc.) it's contribution is 0. We can describe this as a function of time - between 9pm and 10am, for example, a point of the terrain is NOT lit by the sun.
While the sun is fully visible it's contribution is 1. The can happen at between say 10.30am and 8.30pm.
The in between times (as the sun appears over distant peaks) would essentially represent the penumbra period - where we could interpolate, based on time of day, a contribution factor between 0 and 1.
However, whilst the above would work well for the ground, no information would be available to correctly light buildings or other objects.
So my idea is to represent sun light using two height values per point on the terrain (probably per vertex in a heightmap). The first value is the height at which the sun begins to fully light a point above the terrain, and the second value the point below which no sunlight has an effect (the first value would always be higher than the second).
So a fragment shader would effectively be:
if fragment is above first value, sunlight contribution = 1;
else if fragment is below second value, sunlight contribution = 0
else interpolate between the two.
This contribution would be included in a g-buffer (after a depth only pass).
The sun changes position slowly over time, so I think it's perfectly possible to recalculate new values by dealing with a fairly small sector of the 'sunlightmap' each frame and storing the results in a 16-bit RG texture.
I haven't really fleshed out this idea, but I thought I'd throw it out there and hopefully get some feedback.
Cheers.