Yeah, that is somewhat noticeable.   This is a normal kind of design hurdle though.  Not in the least a "show stopper", turn the lights down until the effect isn't nauseating, and continue on.  Come back to it when you have time to make things look better.  I'm sure there's a setting you can adjust or something to make it a non-issue.  You shouldn't let these things become burdensome to your development process, it's not really even a development issue.   My philosophy, which I fail to follow a lot, is to make it work first, make it pretty later.  It works for me when I stick to it. 

I think it looks truly amazing, regardless of the normal shift.

If you can do a gradual change and make it work that would be good. Either way make do with what you have and if it's not 'perfect' oh well... As artists we have that problem where we will work something to death.  I also have a lot of imperfections on my stuff for this challenge too, but I'm letting it go.

I'm not sure if you're doing this in Blender, but if you want me to take a look at it, I can try to find a solution.  

Maybe to make the depth effect more subtle you could try adding some ambient lighting - it might be able to help dampen the effect by adding some additional shadows that don't get swapped.

Alternatively, while it will hurt the brick's depth effect, if you reduce the intensity/height of the mapping it should also reduce it a little at the cost of the bricks looking flatter.

Overall I don't think it looks that bad really, I mean if you take a dark room with shadow casting things and you move a light from side to side, the shadows do move a LOT in a very similar fashion. I think it's a slightly natural effect that is less commonly noticed because we don't often have lights flashing on/off from differing angles that often.

I opened it in gimp to try and figure out what was going on, in the gif the normal map lighting does change instantaneously as you said. I made it into a gif with it blending gradually between the 2:

When I look at the specular highlights around the right hand firepit, one set seems to be moving one way and one the other, as if it doesn't match up with the fire from the pits so it isn't working with our brains maybe?

Thanks for the advice, everyone.

I'll try it out, although latel down the line, of course, otherwise I suspect I'll get into the habit of tweaking tiny details (like the optimum amount of normal mapping) before I've got the big details down... If I can. It's already hard to stop thinking about all the possibilities.

18 hours ago, Rutin said:

I'm not sure if you're doing this in Blender, but if you want me to take a look at it, I can try to find a solution.  

I took the gif already in Unity (if that's what you mean). I actually made the normal maps with a program called "Materialize"... which might be part of the problem, since it doesn't work with real geometry. But there's little that can be done on the Blender side, I think.

It just a kind of optical illusion. It caused not by errors on algo implementation. It comes from human's vision features 

19 hours ago, supermikhail said:

I actually made the normal maps with a program called "Materialize"... which might be part of the problem, since it doesn't work with real geometry.

I think this is probably the root of the problem, it is very difficult to come up with a good normal map from an image, maybe especially a painted image. I tried it also in my 3d paint app just using the pixel colour values for height and it also gives a slight 'moving' effect as you move the light source, although not with the contraction and expansion you are seeing, so that may be partly down to the result from materialize.

For this kind of thing you really want a painting app where you can lay down physical layers of paint so it stores a heightmap as well as albedo. I did actually start writing one years ago but never did anything with it, maybe there are some other programs which do this.

The problem may also be connected to how I drew the tiles. Namely, the diffuse map already kind of simulates bumps with tone in a cartoony way (and Materialize is built with a photorealistic base in mind). It's the first texture I drew for the project (been improving as fast as I can), and I guess it's going to plague me until the end, because at this point it's kind of late to change it, because I've reused it as a base for many (tileable) variations. Unless at the end I have time to redraw probably a dozen textures, if not more.?

Also, if I have to stick with this texture, I guess it's worth looking into flat shading, after all. Of course, my textures have been getting less cartoony...

Well, I've been learning a whole bunch of things on this journey.