Hi Syranide! Thanks for the reply!
You've spotted the problem precisely! The XY-scaling doubles as the LODs get farther away. My ScaleFac variable doesn't take it into account. Unfortunately the error was only in the problem description I've posted here. My code snippet was wrong. Sorry! In fact I calculate the ScaleFac as
Anyway, as for your actual problem, it seems like you forgot to consider the X/Y-scaling of the different heightmap LODs as you compute your normals... remember, one LOD further away, means each quad is twice as wide (but not twice as high!), which also means you have to consider this when computing your normals.
uniform float2 ScaleFac = float2(-0.5/2^L,-0.5/2^L); // L is the clipmap index increasing for the LODs farther away
For the posted image I'm drawing 4 LODs with the squares of the closest LOD being 1x1 units big (2x2 units for the next LOD, than 4x4 and finally 8x8).
I'm not sure how to combine this LOD method with normal maps. As the LOD heightmaps are calculated at run time on the GPU with only incremental updates for being provided by the CPU as the camera moves. Normal-mapping the LODs with the double or tripple resolution would require some efficient update scheme. Right now I'm just trying to get the basic method work.
First off, I would recommend that you implement your normals using textures instead, this will allow you to vastly improve the graphical quality of your terrain, without increasing the vertex count. You can easily use normal maps with 2-3 times higher resolution (than the heightmap), and it's really really cheap.
The LODs don't get finer, only coarser. I'm drawing 4 LOD rings. For each successive ring the size of the squares is doubled. They may seem to get finer but that's just the screenshot.The incorrectly lit area is exactly at the border between 2 rings.
EDIT: I'm not sure what's going on in your wireframe-screenshot... it look likes the LODs get coarser and coarser, and suddenly, some of them get finer... and then coarser again?... those look like the LODs that are incorrectly colored as well.
Anyway after you hint I've double checked my normal scaling factors. They seem to be correct but the problem is not solved. I'm still looking for help.
Using textures for normals is really simple, just treat them as you do with the heightmap... of course there are some things to take care of, such as there being additional LODs for normals and normal LODs "moving faster". But performance shouldn't really be a problem, I did a naive implemention myself for this, and while I could see the CPU spiking with really high detail normal maps being streamed to the GPU. That had a lot more with the unrealistically fast flying movement I had, which may not be obvious at first, my issue really was because I was literally streaming tens of megabytes per second because I was moving so fast (although it didn't seem like it).
Anyway, your choice, but I really would recommend them in the long run... if you want to keep it simple for now, you could just copy the heightmap code and use the same resolution for heightmaps and normalmaps (literally copy-paste, although you need to compute the normal maps). You could even bake the heightmap and normalmap into the same texture, although, it would only be a temporary solution.
Also, looking at the incorrectly colored image above... something else seems amiss... I seems to me that there are two distinct blue colors up there, not one. So the normals seems correct (the shading of it all), but it seems to transition between two different colors? Or rather... like the it transitions from some mixture of RGB, to just B. Messed up the RGB-components somewhere?
Additional info: normal maps calculated for all LODs. The X and Y components are encoded in the red and green color channels. The Z component is supposed to be always 1.
It is obvious that all the textures except the first one have a strange discontinuity along the border. The reason for this is still not clear to me.
I would assume the discontinuity is because you are computing the normals, but you don't actually sample outside of the boundaries specified by the target texture (as is required)... although, the pattern on the edges doesn't really seem ordinary... perhaps the edge normals get computed from some irrelevant data outside the texture? I'm not sure if you've noticed it, but the edge at the top and the bottom are equal or near equal... it would seem like the edges wrap around when sampling the normals?That would also explain why the first image looks correct, because it tiles.