You need to divide dist by ZFAR

float dist = length(v_pos) / ZFAR; 

I also see a minor error: you defined nearPlaneHS as a float, when it should be a vec2, but it wont make or break it

vec2 nearPlaneHalfSize = vec2(screenSize.x / screenSize.y, 1.0) * halfTan;

1. Check that the value of getDepth(refcoord) is constant for all positions in the scene

this is the most likely issue, that your depth texture is wrong somehow

2. Check that length(v_pos) is constant as you move around

I seriously doubt this one is wrong, but you never know

In both cases:

1. If you rotate your camera and move around, the color should only vary depending on distance from camera

2. The values should only range from [0, 1]

3. The values should be completely linear

Once you have verified that one of your inputs are simply wrong, you'll need to figure out the why

Eg. is the values of ZNEAR/ZFAR correct?

Okay, fixed 2 first problems that you mentioned and this is what I got: http://scrapeshare.com/?vid=1411064900979-cqexqe Now there is something like gradient, far fragments is white, near gray, but that's not what I want..

1. Check that the value of getDepth(refcoord) is constant for all positions in the scene

this is the most likely issue, that your depth texture is wrong somehow

I think that depth texture is good, I tested it by rendering plane with depth texture bound to it and it was good, if you need I can post screenshot.

2. Check that length(v_pos) is constant as you move around

I seriously doubt this one is wrong, but you never know

I actually didn't know how I could test it, but I set FragColor = vec4(dist); and here is the result: http://scrapeshare.com/?vid=1411065277580-rx2foy And if dist is value between 0.0-1.0, then I think it's good, all fragments till 1.0 distance has transparency, and which are more far than 1.0 then they are only white, because alpha and color >= 1.0.

Finally.. After so much days of googling I found the solution. Now I realized that I need to have both scene depth and water depth values linear, then subtract them and scale. Here is the code if someone needs it:

vec2 textureCoords = gl_FragCoord.xy / ScreenSize;
float sceneZ = texture2D(sceneDepthTexture, textureCoords).x;
float linearSceneDepth = (2.0 * ZNEAR) / (ZFAR + ZNEAR - sceneZ * (ZFAR - ZNEAR));
float linearWaterDepth = (2.0 * ZNEAR) / (ZFAR + ZNEAR - gl_FragCoord.z * (ZFAR - ZNEAR));
const float alphaScale = 1000.0;
float alpha = clamp(linearSceneDepth - linearWaterDepth, 0.0, 1.0) * alphaScale;

Those depths are in linear window-space, and will have incorrect depths on the edges of the screen.

Regardless, glad you made something work, finally :)