what should be second parameter?

The direction of the light.


L. Spiro

This one:

if(dot(norm, s) < 0.05)

didn't helped, but this:

if(max(dot(norm, s), 0.0) < 0.05)

helped, until I didn't looked at other places, for example top of the hill was not lit by the light, where it should. So this is not the right way as a solution.

abs(dot(norm, s)) < 0.05

This problem seems to be unique to you. You seem to have custom preferences on what should cast shadows and what should not, as this is not-at-all standard practice and will definitely cause artifacts with moving suns.


L. Spiro

I think I should ignore this thing (do not lit surfaces which normals are orthogonal to the light). As you said this problem seems unique to me, so yeah I found out that my bias matrix is wrong, instead of this:

0.5 0.0 0.0 0.0
0.0 0.5 0.0 0.0
0.0 0.0 0.5 0.0
0.5 0.5 0.5 1.0

I've had this:

0.5 0.0 0.0 0.5
0.0 0.5 0.0 0.5
0.0 0.0 0.5 0.5
0.0 0.0 0.0 1.0

But after correcting bias matrix everything got screwed up, and now I can think of one problem, matrices multiplication, I think I did it wrong. My program is written in Java and using LWJGL library, there is a matrix class with multiply function and I think I'm multiplying it wrong. By reading shadow mapping tutorial, my light's matrix should be computed like this:

lightMatrix = bias * projection * view * model

Function for multiply looks like this:

public static Matrix4f mul(Matrix4f left, Matrix4f right, Matrix4f dest) 

It multiplies right matrix by left matrix and stores to the dest matrix.

For testing purposes I set light projection matrix to perspective, and this is how I multiply them():

Matrix4f.mul(projectionMatrix, biasMatrix, lightViewProjectionMatrix);
Matrix4f.mul(viewMatrix, lightViewProjectionMatrix, lightViewProjectionMatrix);

//After every model transform I send this matrix to the shader:
Matrix4f matrix = new Matrix4f();
Matrix4f.mul(modelMatrix, lightViewProjectionMatrix, matrix);

And the only thing that I get is just black line across the terrain

EDIT: It's not the bias problem, I tried to calculate matrices in vertex shader to make sure bias matrix is correct and results were same.

abs(dot(norm, s)) < 0.05

This problem seems to be unique to you. You seem to have custom preferences on what should cast shadows and what should not, as this is not-at-all standard practice and will definitely cause artifacts with moving suns.


L. Spiro

As I said bias wasn't the problem, so I tried this, and still no changes.

It sounds like something bigger is wrong.. Perhaps the vectors you are dotting are not in the same coordinate space?

A good way to debug lighting is to render debugging colors that clearly show what is happening.

Make a lighting debug shader (or mode) which renders green for lit surfaces, red for surfaces backfacing the light, and yellow for surfaces lit but shadowed by the shadowmap.

P.s. "orthogonal" is not the right word for surfaces backfacing the light.. They are just unlit or backfacing. Orthogonal vectors are vectors with no commonality. Perpendicular vectors are orthogonal. For example, the unit x and unit y vectors are orthogonal.

Light's direction vector is in eye space, and normal vector I think is also in eye space, because in vertex shader it's multiplied by normal matrix and normalized, so they are in same coordinate space.

I'm actually not sure how to debug lighting, but I tried things that you said. Here is the screenshot how diffuse and ambient lighting looks (red is ambient, pink is diffuse): http://s18.postimg.org/f2zv74a15/ss09_01_11.png and here is lighting with shadow mapping: http://s23.postimg.org/vlywtvg2j/ss09_01_13.png

The only thing I know, is that the part where is diffuse lighting between shadow and ambient lighting, is the part where lighting transitions from ambient to diffuse and this is the problem, but I dunno how to fix this problem.

Something is wrong with either your terminology, or the way you are thinking about lighting.

All lit surfaces include ambient/emissive and diffuse and specular lighting terms. Unlit surfaces receive only ambient/emissive terms.

Your terrain screenshots would be easier to understand with wireframes, and with more contrasting colors used. (use red for lit, green for unlit)

If you want more detailed help, post a link to your code..

Okay, here is screenshots with wireframes, and more contrasting colors, lit surfaces red, unlit green: http://s11.postimg.org/fj7kr4b1v/ss01_41_07.png and here is shadow mapping, shadowed surfaces black, and not shadowed white: http://s14.postimg.org/4gfhevan5/ss01_40_56.png

I would give you all project files, but it would be mess for you, because you wouldn't know where what is. I can give you some parts of code e.g. lighting and shadow mapping including shaders. Or you could connect to my pc via teamviewer, and I would show you where all needed code is.

P.s. I was thinking about shadow mapping technique that I'm using. In my previous project (accidentally cleared my whole hdd and lost it..) I used different shadow mapping technique and as far as I remember shadow mapping for monkey head were pretty nice, and now I added monkey head to current project and it doesn't looks good. Maybe that's the problem? Currently I do shadow mapping with these texture parameters:

glTexParameteri(GL_TEXTURE_2D, GL14.GL_TEXTURE_COMPARE_MODE, GL_COMPARE_REF_TO_TEXTURE);
glTexParameteri(GL_TEXTURE_2D, GL14.GL_TEXTURE_COMPARE_FUNC, GL_LEQUAL);

So in fragment shader I can use sampler2DShadow and this is how I render shadows

float shadow = textureProj(ShadowMap, ShadowCoord);
FragColor = diffuse * shadow + ambient;

And in previous project it was done differently. I didn't used those texture parameters and instead of sampler2DShadow I used sampler2D. It was actually code from Fabien Sanglard, this article: http://fabiensanglard.net/shadowmapping/ Maybe I should try switching my shadow mapping technique to that, which Fabien Sanglard showed?

Omg.. In blender you can set shading to smooth or flat, I've been using smooth shading because it looked better when lit, and that caused the problem. Here is how it looks with smooth shading: http://s14.postimg.org/6j0rtch1t/ss03_23_39.png and flat: http://s9.postimg.org/55a6z76rj/ss03_23_42.png With flat shading shadow looks better, but how then I would make surfaces look smoother without smooth shading?

It's pretty much impossible for us to understand static terrain screenshots lighting like that. They need to have wireframes, or texture, or a video and motion -- something to help us visually understand them, and we need to know how they are being rendered. (just forward lighting? forward + shadow mapping?)

I don't understand why the flat shaded one looks better. Because the small shadowmap-acne artifact disappears?

If you have a blender file, then it sounds like what you are rendering is just a mesh (not a generated heightmap terrain). Have you tried rendering it in blender? You could set it up for GLSL viewport rendering, or even render it raytraced to get an idea what it should look like.

Can you share the blender file you are using? Perhaps that would clarify what is going on.