Hi guys, thanks again for feedback!
The actual problem I am trying to solve is to make a visibility map for line of sight. Basically you have an observer which can be dragged about on a 2D map and you want all the areas that are visible from the observer's position to be visualized.
In the following screen shot you can see one example where it works more or less as planned with the observer on a mountain, The green areas are visible, while the red are invisible.
LineOfSight.png 392.66KB 3 downloads
In the next shot, I have placed the observer 5 meters above the sea.
LineOfSight2.png 31.38KB 4 downloads
Here you can see we get a cutoff at a couple of kilometers distance. I suspect this is due to the depth map resolution as described in the original post. Another effect I don't understand is the peculiar "pillow" shape of the visibility map. This shape only appears when I account for the earth's curvature in the calculations, but I can't see why it should be shaped like this, it should be circular and further out from the observer. If I do not account for earth's curvature, the shape is more square.
I have tried with non-square projection matrices. The results are much better when I reduce the vertical FOV, but I still don't get the full range, even with the FOV at 5 degrees.
I don't think cascading shadow maps would help in this case. IIRC, in CSM you split the view frustum and make one shadow map for each, section along the distance of the view, but in this case I would be more interested in splitting the light frustum.