I think I have managed to pretty much fix it by converting light's position to eyespace(?) by multiplying modelView matrix by light position in the vertex shader and then outputting it to fragment shader. Now it looks properly, like it's supposed to, but I still want some clarification why is it so.