[EDIT] Thanks to Ashaman, now have PCF edges to the shadows as well.
I'm staying away from stencil shadows after my last attempt and am just doing standard shadow mapping here, using an orthographic projection for the depth rendering as I want sort of cartoony, always from above shadows for this game. It actually works suprisingly well. I do nine samples in the pixel shader and average the result to get the depth at each pixel. Produces very nice results if the depth texture is 2048 x 2048, and passable if it is 512 x 512.
Had to rewrite a bunch of stuff in the Scene and SceneItem classes to allow for rendering different passes, at the moment just so I can render the depth pass, then the main pass, but this will come in handly later on I expect for other fancy effects. You can also now easily create an object that casts no shadow, just by setting its RenderPass::Depth method to RenderMethod::Null and so on.
Had to abandon shadows being casted onto the player character as looked a right mess. One thing about orthographic, from above shadows is you can't have them cast on vertical walls either as it looks wrong, so I pass the normal of the pixel into the shadow pixel shader, and only shadow it if its dot with 0, 1, 0 is above a certain threshold.
Since journals are always better with images, here is my depth texture for you to enjoy.
Thanks for reading.
Great work !
Are you fetching 9 depth values, create the avg depth from it and compare it vs the scene depth ? The correct way would be to fetch 9 shadow values (comparision already done, result is 0=out of shadow, 1 = in shadow) and average this result, resulting in some soft shadow edges (PCF).
A nice,easy improvement is to start with 4 shadow-fetches and only continue to fetch (eg) 16 more if not all of the original 4 are either completly in shadow or out of shadow.