mikiex

Well there is no reason to be blending the targets alpha SrcBlendAlpha DestBlendAlpha ?

I'm not saying thats the problem though, but thought I would point it out.

These are normally set to one or zero.

To further what  Mhagain is saying:

You might want to also consider doing a distortion in screen space, or world space

You could:

input a float3 constant of the world position you want the centre of the effect from into a shader.

In the shader use the world position of a pixel (you can work this out in the vertex shader and output it to the pixel shader)

Now you can do some sort of effect in worldspace.

For instance (ugly - pseudocode! - some things maybe the wrong way around!):

in the pixel shader:

WorldGradient = distance(CentreOfEffect,WorldPixelPos) //a float result of  a gradient in worldspace

ripple = sin((WorldGradient*Freq)+(Time*Speed)) //sine wave out from the centre of  your effect that moves with Time (input that as a constant)

ripple = ripple*gradient //fade off effect

//So now you have a greyscale ripple in worldspace, combine this into the texture coords of the texture you want to distort

texture = tex2d(tex.xy+ripple,sampler)

//Freq is whatever looks good, this could be multiplied in some way by the distance to make the ripples get larger as they go out.

The brushes using these blend modes in photoshop is the same as layer blend modes, but using the brush colour instead. They can be quite destructive so you might be better implementing them as layers. Search google for:

equation dodge burn

You should find plenty of info, the layer blend modes have been documented many times.

mikiex,

 

Thaks for clarification, but I want that camera will be as close to object as possible. Do you know something about logarithmic depth buffers. I tried to find an example for DX 11, but I din`t find any code example for it. 

Close as possible is your eyeball touching the surface a near clip of 0 I don't know anything about log depth buffers other than the concept exists, certainly DX11 would make things like this possible I am sure. Consider though not many games in the past have bothered to come up with a physically correct solution, yes its been a matter of contention but every project I've worked on we have managed to work around this issue by faking it.

Hm, looks like this was the reason. I set 

screenDepth = 60000.0f;
screenNear = 1.0f;

and everything seems OK, but I cannot understand how in that case render a big/huge maps, for example, for a hundred of thousands km of space? Because of screen depth it cannot be rendered. What I must do in that case?

It's worth pointing out the depth buffer being normally non-linear,  changing the near clip makes a huge difference compared to moving the far.

So you want to push the near as far forward as possible, but this depends on how close you camera will be to objects in the scene. The far does matter, but getting the near correct is most important. Also remeber you say 3km, but you mean 3k of unitys as scale is arbitrary.