How is the sceneDepth texture generated? Isn't this already linear?

Why are you taking the length of the water vector? You only need the Z component.

This is how I generate sceneDepth texture:

depthTexture = glGenTextures();
glActiveTexture(GL_TEXTURE5);
glBindTexture(GL_TEXTURE_2D, depthTexture);
ARBTextureStorage.glTexStorage2D(GL_TEXTURE_2D, 1, GL14.GL_DEPTH_COMPONENT32, Display.getWidth(), Display.getHeight());
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
		
depthFbo = glGenFramebuffers();
glBindFramebuffer(GL_FRAMEBUFFER, depthFbo);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_TEXTURE_2D, depthTexture, 0);
glDrawBuffer(GL_NONE);

So if I don't linearize scene depth, and take length only from Z component, code becomes this:

vec2 coord = gl_FragCoord.xy / ScreenSize;
float vWaterDepth = length(ViewCoord.z);
float vSceneDepth = texture2D(sceneDepth, coord).x;
		
color.a = clamp((vSceneDepth - vWaterDepth) * 1.0, 0.0, 1.0);

And result: http://imageshack.com/a/img674/2048/kzOGM9.jpg water is only 1.0f-2.0f radius circle around camera.

So water depth equals to length(ModelViewMatrix * VertexPosition) ? Okay I did this, then I linearize the depth which is got from scene depth texture:

const float f = 1000.0;
const float n = 0.1;
	
vec2 coord = gl_FragCoord.xy / ScreenSize;
float vWaterDepth = length(MVPos.xyz);
float vSceneDepth = (2 * n) / (f + n - texture2D(sceneDepth, coord).x * (f - n));
		
color.a = (vSceneDepth - vWaterDepth) * 1.0; //Subtract scene depth from water depth and scale the result

And then I can't even see water, it's invisible.

length(MVPos) has to be divided by zfar.

It's also possible vSceneDepth is [0, zfar] instead of [0, 1], so try dividing.

You can have a look at this one, but its a production shader so nothing in there is educational. If it helps, good, if not, then I guess ask:

https://github.com/fwsGonzo/cppcraft/blob/master/Debug/shaders/blocks_water.glsl

So water depth equals to length(ModelViewMatrix * VertexPosition) ? Okay I did this, then I linearize the depth which is got from scene depth texture:

const float f = 1000.0;
const float n = 0.1;
	
vec2 coord = gl_FragCoord.xy / ScreenSize;
float vWaterDepth = length(MVPos.xyz);
float vSceneDepth = (2 * n) / (f + n - texture2D(sceneDepth, coord).x * (f - n));
		
color.a = (vSceneDepth - vWaterDepth) * 1.0; //Subtract scene depth from water depth and scale the result

And then I can't even see water, it's invisible.

length(MVPos) has to be divided by zfar.

It's also possible vSceneDepth is [0, zfar] instead of [0, 1], so try dividing.

You can have a look at this one, but its a production shader so nothing in there is educational. If it helps, good, if not, then I guess ask:

https://github.com/fwsGonzo/cppcraft/blob/master/Debug/shaders/blocks_water.glsl

If I divide both, then water is invisible, if only length(MVPos) then water alpha is 1.0.

I am still not really sure how you get the scene depth texture. Do you copy the depth buffer? I have the feeling that you are mixing to different ways of doing the distance calculation.

In the attached image there are two ways how this could be done. On the left side is the physically correct way. On the right side is a hack that is a bit faster but produces wrong results.

Like I wrote. Choose one but don't mix them up.

I am still not really sure how you get the scene depth texture. Do you copy the depth buffer? I have the feeling that you are mixing to different ways of doing the distance calculation.

In the attached image there are two ways how this could be done. On the left side is the physically correct way. On the right side is a hack that is a bit faster but produces wrong results.

Like I wrote. Choose one but don't mix them up.

For scene depth I render scene to the depth texture with fbo after rendering refraction texture, clip plane is disabled while rendering depth texture. And then I render water plane to the depth texture, to get water depth. And if I get depth from the depth texture using this code:

vec2 coord = gl_FragCoord.xy / ScreenSize;
float vWaterDepth = texture2D(waterDepth, coord.xy).x;
float vSceneDepth = texture2D(sceneDepth, coord.xy).x;
	
vec4 color = refractionColor + reflectionColor + specular;
	
color.a = clamp((vSceneDepth - vWaterDepth) * 1.0, 0.0, 1.0);

Result looks like this: https://imagizer.imageshack.us/v2/640x480q90/540/Dfkzrr.jpg

And you said to use length function to get depth, but.. How I would get terrain depth? I only render water plane and there is VertexPosition attribute which is water's vertex position, and how I would get terrain vertex, to get it's depth using length function?

You can reconstruct the fragment position from the depth value:

http://stackoverflow.com/questions/17292397/reconstruct-fragment-position-from-depth

You can reconstruct the fragment position from the depth value:

http://stackoverflow.com/questions/17292397/reconstruct-fragment-position-from-depth

So code should look like this?

float vWaterDepth = length(ModelViewPosition.z);
	
vec4 TerrainVertexPosition = vec4(0.0);
TerrainVertexPosition.x = gl_FragCoord / ScreenSize.x;
TerrainVertexPosition.y = gl_FragCoord / ScreenSize.y;
TerrainVertexPosition.z = texture2D(sceneDepth, TerrainVertexPosition.xy).x;
TerrainVertexPosition.w = 0.0;
vec3 xyz = (InverseProjectionMatrix * TerrainVertexPosition).xyz;
	
float vSceneDepth = length(xyz.z);
float alpha = (vSceneDepth - vWaterDepth);
	
vec4 color = refractionColor + reflectionColor + specular;
color.a = alpha;

Result: https://imagizer.imageshack.us/v2/640x480q90/674/kzOGM9.jpg

Modestas, I think it would be better if you remove everything except the alpha calculation from the shader for now. Maybe even output alpha as grayscale so you can better focus on the problem. Currently it's really difficult to see what's wrong. Maybe even output the depth value as color for debugging purposes

the value you want I simply called wdepth, look how simple it is:

// read underwater depth

float wdepth = getDepth(refcoord) - dist;

where dist is the distance to vertex from eye... and for some reason you only calculate length() of the scalar.z which... is the same as doing sqrt(z*z) == z.

dist = length(v_pos);

Btw. you cant send "dist" in my example as input to fragment shader, because it doesnt interpolate well on stuff not in a grid pattern,

instead you need to send vec3 viewPos, and then take the length in the fragment shader.

----------------

Lets keep things simple and forget all about the window-space-ness of the depth-value and just use it directly.

refcoord here is the texture-coord of as seen on-screen. I hope that is abundantly clear, as you are reading a screen-space texture.

vec2 refcoord = gl_FragCoord.xy / screenSize.xy;

from this we can read from out texture:

float depth = getDepth(refcoord);

float getDepth(in vec2 uv)
{
   // exp. depth in window-space
   float wsDepth = texture(depthtexture, uv).x;
   // linear depth in window-space
   wsDepth = ZNEAR / (ZFAR - wsDepth * (ZFAR - ZNEAR));
   
   // this converts it to eye-space (but lets ignore that for now, since the value is almost
   //     the same anyways, google nearplanehalfsize after you get the reconstruction working)
   //return wsDepth * length(vec3((uv * 2.0 - vec2(1.0)) * nearPlaneHalfSize, -1.0));
   return wsDepth;
}

You should now have calculated linear depth that is good enough to prove that this works with.

What you should be able to see if you use the depth difference correctly is that when you rotate the camera that depth changes "slightly" on edges of the screen on the left and right, as well as on the top and bottom. This is because the real eye-vector forms a small dome: The distance to a point on a Z plane are not constant.

Now that you have the depth difference, you need to use it for something useful.

The difference value is for example pretty small, so you need rescale it to fit your water

for example:

float deep = min(1.0, wdepth * 16.0);

color = mix(shallowColor, deepColor, deep);

Some proof here: http://fbcraft.fwsnet.net/proof.png

Calculation for nearPlaneHalfSize:

// calculate half near-plane size
const double pio180 = 4.0 * atan(1.0) / 180.0;

float halfTan = tan(fov * pio180 / 2.0);
nearPlaneHalfSize = vec2(halfTan * aspect, halfTan);

the value you want I simply called wdepth, look how simple it is:

// read underwater depth

float wdepth = getDepth(refcoord) - dist;

where dist is the distance to vertex from eye... and for some reason you only calculate length() of the scalar.z which... is the same as doing sqrt(z*z) == z.

dist = length(v_pos);

Btw. you cant send "dist" in my example as input to fragment shader, because it doesnt interpolate well on stuff not in a grid pattern,

instead you need to send vec3 viewPos, and then take the length in the fragment shader.

----------------

Lets keep things simple and forget all about the window-space-ness of the depth-value and just use it directly.

refcoord here is the texture-coord of as seen on-screen. I hope that is abundantly clear, as you are reading a screen-space texture.

vec2 refcoord = gl_FragCoord.xy / screenSize.xy;

from this we can read from out texture:

float depth = getDepth(refcoord);

float getDepth(in vec2 uv)
{
   // exp. depth in window-space
   float wsDepth = texture(depthtexture, uv).x;
   // linear depth in window-space
   wsDepth = ZNEAR / (ZFAR - wsDepth * (ZFAR - ZNEAR));
   
   // this converts it to eye-space (but lets ignore that for now, since the value is almost
   //     the same anyways, google nearplanehalfsize after you get the reconstruction working)
   //return wsDepth * length(vec3((uv * 2.0 - vec2(1.0)) * nearPlaneHalfSize, -1.0));
   return wsDepth;
}

You should now have calculated linear depth that is good enough to prove that this works with.

What you should be able to see if you use the depth difference correctly is that when you rotate the camera that depth changes "slightly" on edges of the screen on the left and right, as well as on the top and bottom. This is because the real eye-vector forms a small dome: The distance to a point on a Z plane are not constant.

Now that you have the depth difference, you need to use it for something useful.

The difference value is for example pretty small, so you need rescale it to fit your water

for example:

float deep = min(1.0, wdepth * 16.0);

color = mix(shallowColor, deepColor, deep);

Some proof here: http://fbcraft.fwsnet.net/proof.png

Calculation for nearPlaneHalfSize:

// calculate half near-plane size
const double pio180 = 4.0 * atan(1.0) / 180.0;

float halfTan = tan(fov * pio180 / 2.0);
nearPlaneHalfSize = vec2(halfTan * aspect, halfTan);

Still nothing.. By reading what you've said this is what I got (I removed all unnecessary things such as refraction, reflection and left only deep color and shallow color):

const float ZFAR = 1000.0;
const float ZNEAR = 0.1;

float getDepth(in vec2 uv)
{
	float wsDepth = texture(sceneDepth, uv).x;
	wsDepth = ZNEAR / (ZFAR - wsDepth * (ZFAR - ZNEAR));
	
	const float pio180 = 4.0 * atan(1.0) / 180.0;
	float halfTan = tan(45.0 * pio180 / 2.0);
	float nearPlaneHalfSize = vec2(halfTan * 1.3333334, halfTan);
	return wsDepth * length(vec3((uv * 2.0 - 1.0) * nearPlaneHalfSize, -1.0));
}

subroutine(RenderPassType)
void drawWater()
{
	//texture-coord of as seen on-screen
	vec2 refcoord = gl_FragCoord.xy / ScreenSize;
	//dist is the distance to vertex from eye
	//v_pos = -(ModelViewMatrix * VertexPosition).xyz;
	float dist = length(v_pos);
	//read underwater depth
	float wdepth = getDepth(refcoord) - dist;
	//scale value
	float deep = min(1.0, wdepth * 16.0);
	
	const vec3 deepColor = vec3(42, 73, 87) * vec3(1.0 / 255.0);
	const vec3 shallowColor = vec3(0.35, 0.6, 0.45);
	FragColor = vec4(mix(shallowColor, deepColor, deep), 1.0);
}

And only thing what I can get is the circle which moves when I rotate/move camera: http://imageshack.com/a/img911/8084/mDB8ac.jpg And when camera moves higher (on Y axis), it disappears.