Almost immediately after posting, I found a solution; however, I don't understand why I works.
I changed the line:
normalMap = normalMap * 2 - 1; // Normalize to [-1, 1]
normalMap.xy = normalMap.xy * 2 - 1; // Normalize to [-1, 1]
So that the z value would stay in the range [0,1]. Now it looks like this:
If someone knows why that change worked and if it while cause problems later, I would be grateful for the insight.