Screenspace Normals - Creation, Normal Maps, and Unpacking
How are screenspace normals created, and is this step before or after using normal maps or bump maps? If it's done before using normal maps, how are normal maps going to be affected by the screenspace-ness of the normals, and if after, how can you justify using a Model-View matrix on all fragments rather on vertices? Isn't that a lot more calculations?
Finally, how are they unpacked? I realize you can get the blue component using pythagoras, but how are they returned to world space?
After unpacking, you only have to rotate them into world space (using the view_to_world matrix) if all your lighting data is in world space. If you put all your data in view space, this isn't an issue.
You can also just store two component normals in world space if you want to.
You actually can't just use Pythagoras to decode n.xy into n.xyz because that gives two two answers: +z and -z. You might think that this is fine because view-space normals will always point towards the camera, but *due to perspective*, this is false. It's very easy for a surface to be visible yet facing away from the camera (just look down a corridor where the floor slopes downwards)!
So, this simple packing scheme of "just drop z" actually must be "replace z with one bit" so you know which of the two decoding to use.
This page has a good listing of a variety of two component encodings:
http://aras-p.info/texts/CompactNormalStorage.html
My current favourite is octahedral:
https://knarkowicz.wordpress.com/2014/04/16/octahedron-normal-vector-encoding/
You might think that this is fine because view-space normals will always point towards the camera, but *due to perspective*, this is false. It's very easy for a surface to be visible yet facing away from the camera (just look down a corridor where the floor slopes downwards)!
I believe the problem is not perspective itself, but rather because we cheat. "Smooth" per-vertex normals instead of sharp per-face normals. Just look at this picture. It's ortho projection, yet one of the normals is facing away from the camera. Would we use "real" per face normals this wouldn't happen. But then it would be extremely hard to make a low tessellated sphere look round.
I believe the problem is not perspective itself, but rather because we cheat. "Smooth" per-vertex normals instead of sharp per-face normals.
Well, it's both :)
I didn't consider that case, but it certainly is an issue, especially when using large polygons and a large "smoothing angle"...
Perspective is a separate cause as well though. Consider a the case where you're looking straight ahead down a rectangular corridor -- your view direction is, say (0,0,1) and the floor's normal is (0,1,0). If you then make the corridor slope downwards, so the opposite end is lower than you, then the floor's flat/face normal is now pointing away from you (something like (0, 0.96, 0.28)), yet is clearly visible.
I know this doesn't address the underlying question, but I thought I'd throw it out there just in case (food for thought). Perhaps You could store the azimuth/altitude of the vector as opposed to the cartesian coordinates, thereby reducing the dimension to 2?
Using spherical coordinates have been brought up several times in the past, and it's included in Aras's comparison. It turns out it's not particular fast to encode/decode, and the straightforward version also doesn't have a great distribution of precision.