You mean like so ?
Didn't seem to help.
And yes I've tried linear sampling, also no help.
UPDATE: saturate made it a bit better, still black spots there when comparing it to just sampling xyz from the normal map.
I'm gonna do that for now, if anyone has an idea please go ahead and tell me