Hello

I have read a bunch of articles and tutorials about normal mapping, many of them focus on how to create them but not that many understandably describe them as they as used in the graphics pipeline.

I understand that the Normals of a high-res model is saved as a pixel in a texture. And applying this texture to a low-res model to get per-pixel normals.

I understand that these Normals are in high-res model space. And for them to work, the low-res model must be aligned with the high-res model (same position, rotation and scaling).

I understand that if the low-res Model is rotated, that the Normals in the normap map will remain in high-res space and there will be a mismatch.

What I don't understand is why do we call it tangent space instead of just "original high-res model space", and why is it continously changing for each fragment? One article says that each normal is in the space of its individual triangle, does that mean that in the high-res model, each face has its own space, and the neighbouring face has a different space? Why not just one space for all faces?

Why can't I just take the light vector, transform it with to model space using the inverse-model transformation matrix?

What I can think of is that, when I animate a character each face will be transformed differently. This means that for each face the normals will be transformed differently, from the bone matrices. Why can't I just take the light vector, transform it to the "face-space" using the inverse-bone matrices, and use the normal map from there on? Why do we involve tangent, bitangent using UVs?

It is all just a big mess to me, if anyone can describe it please do so.

Cheers!

I have a little rant that doesn't do anything to help the OP (sorry). I find the use of the word "space" extremely confusing. There is only one space in this context (3-dimensional Euclidean space), and what we are talking about here are multiple frames of reference or coordinates.

This is particularly annoying because there is a meaningful notion of "tangent space" of a manifold at a point, which could make sense in this situation. But it would be a two-dimensional space, so that's not what we are talking about here.

This took me a while to work out and I find things easier to understand conceptually rather than technically, so if the other answers haven't helped, I can explain how I see the reason why tangent space normal mapping may be required...

If you have "object space" mapping, each normal (pixel) in the normal map represents where it is on the model is was designed for, and hence the exact direction the normal is facing. So if you have a cylinder objects and you wanted to wrap it with an object space normal map, the normal map would not look uniform across it. The area covering the 'front' of the cylinder would look very different to the area covering the back because the normals in the map represent the normals of the 'bumpiness' and also the normals of the object faces themselves. You can use this on objects that don't deform but if the object rotates, you'll obviously need to rotate the light coming in also (or rotate every normal in the map but rotating the light to match the object is quicker). These types of normal maps are generally only suited to the specific objects they wrap (or at least same shape objects) because the normals of the object are 'built in'.

"Tangent space" normal maps look different. They only have the normals of the bumpiness of the surface built in, not the normals of the surface itself, so you can effectively apply it to the cylinder just as easily but your calculation would include the surface normal as well as the normal map normal. These generally have a uniform bluish tint across the whole normal map and can be applied to literally any model because they only include bumpiness data. Here's where tangent space comes in. Imagine you're shading a particular pixel, you have the surface normal, you have the normal map normal and you have the ray of light. Now, when the normal map was designed, it was created in, let's say, screen space. So your nice rock texture was facing you on the screen and you carefully set all the normals to be facing directly at you, it looks great. So how do you know which 'orientation' the normal map should be in when you're shading this pixel on an arbitrary model? All you have is the surface normal, the normal map normal (in screen space only, don't forget) and the ray of light. From this data, you have no idea what orientation the model designer put it in. When you originally wrap the object with the normal map, you need to somehow tell each vertex and thence, each pixel, the orientation of the normal map. To do this, you store the two vectors which are perpendicular to the vertex normal which represent the 'up' direction and the 'right' direction which effectively map to the up and right direction in your screen space normal map. This is essentially saying, "at this vertex, the normal map is oriented in this direction", so when you come to do your light calculation, you have an exact reference to know exactly in which direction the normal should be pointing - you know which direction it's pointing in your original screen space, but that's not enough when you're applying it to an arbitrary model.

Feel free to correct me if I'm wrong, but that's how I see tangent space.

So in the first instance, where the Normals are "built in", The sampled normal replaces the normal passed from the vertex shader. This means that only static objects can use this.

When a Normal Map is created for use in Tangent space, the normal stored in the map is different from this "built in" case. The normal stored in the Tangent Space Normal Map is the offset to apply the normal passed from the vertex shader?

Lets keep track of the Normals here, I can count 4 normals which I confuse together:

-Normal passed from Vertex Shader

-Normal sampled in map

-Normal passed from Vertex Shader + some calculation

-Normal sampled in map + some calculation

And there are 3 of these making up a face. So more combinations to get mixed up. Which of these comes in what order and what is done with each of them?

Is this correct:

1. From the vertex shader 3 vertices(with normals) are outputted.

2. In the pixel shader, an interpolated normal (N1) from these 3 vertex normals is passed.

3. A sampled Normal (N2) from the normal map is retrieved.

4. Using (N1) and (N2) we create a transform matrix (TBN).

5. We then use the (TBN-1) to transform the light vector in object space (LM) to Tangent space (LT).

6. We then use either (N1) or (N2) or a calculation of both to DOT it with (LT).

7. Use result to shade the pixel.

What is wrong here?

When a Normal Map is created for use in Tangent space, the normal stored in the map is different from this "built in" case. The normal stored in the Tangent Space Normal Map is the offset to apply the normal passed from the vertex shader?