Even with high resolution like 2049 x 2049, there are aliasing effects.
I'm rendering a terrain with size 2049 x 2049 and with shadows.

The shadow map resolution is not the only important factor, equally important is how large area is the shadowmap covering. Even a shadowmap with 4096x4096 would have extremely huge aliasing if you used it for example for the whole Skyrim map.

So what is the solution to this problem? should we blur shadow map?

As said above, probably your light is to far away from the view point.

Take a look at this. It is called parallel split shadow map:

http://http.developer.nvidia.com/GPUGems3/gpugems3_ch10.html

Are youe using a custom filtering kernel? seems to me you have a bug in the depth comparison method which causes results between neighbouring pixels in the map not to match up.

Are youe using a custom filtering kernel? seems to me you have a bug in the depth comparison method which causes results between neighbouring pixels in the map not to match up.

This was the problem. I forgot to restore the rasterizerState after rendering the scene to shadow map.

Thank you everyone, now the problem is solved. You all are great. (the resolution used in this image for shadow map is 1024x1024 with 2049x2049 terrain.)