This might be stupid error here, but I just don't know where to look anymore...



What you see is directional light VSM. I think I somehow set variance value wrong. But I'm not sure. Here's the code

///////Shadow Map PS
{
OutputPSShadow outgoing = (OutputPSShadow)0;
//I did this so it stays in [0,1] range
float depth = length(incoming.view) / 1000;
float2 moment;
moment.x = depth;
float dx = ddx(depth), dy = ddy(depth);
moment.y = depth * depth + 0.25 * (dx*dx + dy*dy);
outgoing.color.r = moment.x;
outgoing.color.g = moment.y;
outgoing.color.a = 1;
return outgoing;
}
///////Shadow Casting function
{
float2 tex = incoming.mapSamplingPos.xy * float2(0.5, -0.5) + 0.5;
//if not in Shadow map Area then no shadow
if(saturate(tex.x) == tex.x && saturate(tex.y) == tex.y)
{
float disToLight = length(incoming.view) / 1000;
float lit = 0;
//Shadow Map
float2 moment = tex2D(ShadowSampler, tex).xy;
//if moment.x is zero then something's wrong so no shadow
if(moment.x == 0) return 1;
float e_x2 = moment.y;
float ex_2 = moment.x * moment.x;
float variance = e_x2 - ex_2;
float md = disToLight - moment.x;
float p = variance / (variance + md*md);
lit = max(p, disToLight <= moment.x);
return lit;
}
else return 1;
}


I just don't know where to look anymore, please help.

It seems like a lack of precision. What is the format of your linear depth buffer ?

It seems like a lack of precision. What is the format of your linear depth buffer ?

If you mean texture format for the shadow map, it's R5G6B5. I tried changing it to A16B16G16R16 but nothing changed. Or is it because I divide the depth by 1000 so it stays in [0, 1] range?

The linear depth and squared linear depth have to be in the range [0, 1] so dividing view.z by your far plane is necessary.

Here you are trying to store 2 depth data (moment.x and moment.y) so a 2-channel texture format will do.
Depth values need a lot of precision to be stored. In R5G6B5, you use 5 bits for moment.x and 6 bits for moment.y, that is NOT enough.

I'm surprised that a 16-bit format didn't work. It should be enough for a small scaled scene like yours.
I see you don't use a minimum VSM value in your code, have a look at GPU Gems 3 (Example 8-1), it may help.

I set the min. variance value, turns out shadow of small objects like in the picture disappeared along with the strip while a bigger ones stays fine or, at least, they have darker shadow. Is it possible that I set light projection matrix wrong? Current light eye position is camera pivot - (light vector * 500).