# Help, understanding Tangent Space (normal mapping)

This topic is 1734 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hello

I have read a bunch of articles and tutorials about normal mapping, many of them focus on how to create them but not that many understandably describe them as they as used in the graphics pipeline.

I understand that the Normals of a high-res model is saved as a pixel in a texture. And applying this texture to a low-res model to get per-pixel normals.

I understand that these Normals are in high-res model space. And for them to work, the low-res model must be aligned with the high-res model (same position, rotation and scaling).

I understand that if the low-res Model is rotated, that the Normals in the normap map will remain in high-res space and there will be a mismatch.

What I don't understand is why do we call it tangent space instead of just "original high-res model space", and why is it continously changing for each fragment? One article says that each normal is in the space of its individual triangle, does that mean that in the high-res model, each face has its own space, and the neighbouring face has a different space? Why not just one space for all faces?

Why can't I just take the light vector, transform it with to model space using the inverse-model transformation matrix?

What I can think of is that, when I animate a character each face will be transformed differently. This means that for each face the normals will be transformed differently, from the bone matrices. Why can't I just take the light vector, transform it to the "face-space" using the inverse-bone matrices, and use the normal map from there on? Why do we involve tangent, bitangent using UVs?

It is all just a big mess to me, if anyone can describe it please do so.

Cheers!

##### Share on other sites

You can have just one space for all faces (object space), the main advantage of using tangent space is that you can use the same diffuse and normal texture on multiple faces (say a box with the same texture on all faces), and you can do deformation with skinning.

##### Share on other sites

I have a little rant that doesn't do anything to help the OP (sorry). I find the use of the word "space" extremely confusing. There is only one space in this context (3-dimensional Euclidean space), and what we are talking about here are multiple frames of reference or coordinates.

This is particularly annoying because there is a meaningful notion of "tangent space" of a manifold at a point, which could make sense in this situation. But it would be a two-dimensional space, so that's not what we are talking about here.

##### Share on other sites

It's pretty standard terminology in 3d graphics though (World Space, Object Space, Bone Space, Tangent Space, Mighty Morphin' Space Rangers Space), so I guess you'll just have to grin and bearspace it ;)

Probably comes from the notion of a vector space (but as you say really means coordinates of a vector space with respect to a given basis).

##### Share on other sites

I understand that the Normals of a high-res model is saved as a pixel in a texture. And applying this texture to a low-res model to get per-pixel normals.
I understand that if the low-res Model is rotated, that the Normals in the normap map will remain in high-res space and there will be a mismatch.

What you're describing would be a world-space/model-space/object-space normal map, not a tangent-space normal map.

We usually say that vertex normals are in "model-space", the frame of reference that's relative to how the model is oriented in the world. To transform these per-vertex normals to world-space, we multiply them with a model-to-world matrix, to to transform them to view-space, we multiply them with a model-to-world-to-view matrix.

If you encoded these normals directly in a texture, you'd have a model-space normal map, which would behave like you describe -- if you rotate the model, the texture will be wrong (unless you also apply the model-to-world matrix to the texture).

For most meshes, yes you actually can use a model-space normal map if you want to. These only become a problem for skinned meshes like you point out, because every vertex might be transformed differently. What you're calling "face space" is the "tangent space"!

To solve this problem (that the model can't be deformed from it's original pose), we often use tangent-space normal maps. Tangent-space is a frame-of-reference that's defined at every vertex, looking out from that vertex. Each vertex contains a normal (forward/z axis), a tangent (right/x axis) and a binormal (up/y axis). At every single vertex, if you move in the z-axis (of tangent-space), you'll move away from the object. If you move in the x or y axis, you'll move across the faces of the object but never leave it.

Because this space is defined at every vertex, we interpolate it across every triangle, down to every pixel. Three vertices ouput their normal/tangent/binormal (a 3d frame of reference) from the vertex shader, and these are interpolated/blended-together before being fed into the pixel shader. Therefore, every pixel also has it's own unique "tangent-space" frame of reference, where z is away from the surface, and x/y are along the surface. It's as if the tangent-space frame of reference is warped, so that it's oriented differently depending on which vertex/pixel you look at.

A tangent-space normal is defined with regards to this "warped" space. This is why a blue colour is flat in a TS normal map -- (x=0,y=0,z=1) is always pointing directly out of the surface.

To get a model-space vertex, we treat the normal/tangent/binormal as a 3x3 matrix, and multiply it with the vector from the texture.

This is equivalent to:

nmap = tex2d(...)*0.5+0.5

model_normal = nmap.r * tangent + nmap.g * binormal + nmap.b * normal;

As you can see, each channel of the TS normal map is saying how much the normal should point in each of the tangent-space axes.

##### Share on other sites

I think the others (including the OP) have more or less provided the answer - tangent space normal mapping is useful in situations like skinning, where the initial definition of the surface (in what everyone is referring to as model space) is changed dynamically in some way.  Another situation would be if you were applying a normal map to a dynamic surface, such as a water simulation.  If you are working with a static model that won't deform at runtime, then there is no reason that you can't use the method that you are describing and keep the normal vector information from the normal map encoded in model space.  You would just have to transform that normal vector with the same transform for your model to ensure it lines up right.

More to your original question, it is called tangent space because of how the basis for that space is chosen.  Since the tangent and bitangent are perpendicular to the normal vector at all points on the surface of the mesh, it becomes quite natural to call it tangent space - two thirds of its definition are tangent vectors!

##### Share on other sites

This took me a while to work out and I find things easier to understand conceptually rather than technically, so if the other answers haven't helped, I can explain how I see the reason why tangent space normal mapping may be required...

If you have "object space" mapping, each normal (pixel) in the normal map represents where it is on the model is was designed for, and hence the exact direction the normal is facing.  So if you have a cylinder objects and you wanted to wrap it with an object space normal map, the normal map would not look uniform across it.  The area covering the 'front' of the cylinder would look very different to the area covering the back because the normals in the map represent the normals of the 'bumpiness' and also the normals of the object faces themselves.  You can use this on objects that don't deform but if the object rotates, you'll obviously need to rotate the light coming in also (or rotate every normal in the map but rotating the light to match the object is quicker).  These types of normal maps are generally only suited to the specific objects they wrap (or at least same shape objects) because the normals of the object are 'built in'.

"Tangent space" normal maps look different.  They only have the normals of the bumpiness of the surface built in, not the normals of the surface itself, so you can effectively apply it to the cylinder just as easily but your calculation would include the surface normal as well as the normal map normal.  These generally have a uniform bluish tint across the whole normal map and can be applied to literally any model because they only include bumpiness data.  Here's where tangent space comes in.  Imagine you're shading a particular pixel, you have the surface normal, you have the normal map normal and you have the ray of light.  Now, when the normal map was designed, it was created in, let's say, screen space.  So your nice rock texture was facing you on the screen and you carefully set all the normals to be facing directly at you, it looks great.  So how do you know which 'orientation' the normal map should be in when you're shading this pixel on an arbitrary model?  All you have is the surface normal, the normal map normal (in screen space only, don't forget) and the ray of light.  From this data, you have no idea what orientation the model designer put it in.  When you originally wrap the object with the normal map, you need to somehow tell each vertex and thence, each pixel, the orientation of the normal map.  To do this, you store the two vectors which are perpendicular to the vertex normal which represent the 'up' direction and the 'right' direction which effectively map to the up and right direction in your screen space normal map.  This is essentially saying, "at this vertex, the normal map is oriented in this direction", so when you come to do your light calculation, you have an exact reference to know exactly in which direction the normal should be pointing - you know which direction it's pointing in your original screen space, but that's not enough when you're applying it to an arbitrary model.

Feel free to correct me if I'm wrong, but that's how I see tangent space.

##### Share on other sites

So in the first instance, where the Normals are "built in", The sampled normal replaces the normal passed from the vertex shader. This means that only static objects can use this.

When a Normal Map is created for use in Tangent space, the normal stored in the map is different from this "built in" case. The normal stored in the Tangent Space Normal Map is the offset to apply the normal passed from the vertex shader?

Lets keep track of the Normals here, I can count 4 normals which I confuse together:

-Normal passed from Vertex Shader

-Normal sampled in map

-Normal passed from Vertex Shader + some calculation

-Normal sampled in map + some calculation

And there are 3 of these making up a face. So more combinations to get mixed up. Which of these comes in what order and what is done with each of them?

Is this correct:

1. From the vertex shader 3 vertices(with normals) are outputted.

2. In the pixel shader, an interpolated normal (N1) from these 3 vertex normals is passed.

3. A sampled Normal (N2) from the normal map is retrieved.

4. Using (N1) and (N2) we create a transform matrix (TBN).

5. We then use the (TBN-1) to transform the light vector in object space (LM) to Tangent space (LT).

6. We then use either (N1) or (N2) or a calculation of both to DOT it with (LT).

7. Use result to shade the pixel.

What is wrong here?

##### Share on other sites

When a Normal Map is created for use in Tangent space, the normal stored in the map is different from this "built in" case. The normal stored in the Tangent Space Normal Map is the offset to apply the normal passed from the vertex shader?

Yes. The tangent space normal map normals are pointing in the correct direction when your matching texture is facing you on your screen. If you wrap that texture around an object, the pixels/normals you pull out of the normal map in the shader will still be in screen space, so you have to apply them to the surface normal at that pixel.

So in the vertex shader, you'll need to pass the vertex normal, the normal map normal and the tangent/bitangent vectors to the pixel shader. The tangent/bitangents vectors coupled with the vertex normal is enough to orient the normal map normal [technical details skipped] to the correct vector. You can then use this in your lighting calculation.

One thing I was wondering actually, if the normals in the normal map always point generally 'out' of screen space, why do you need both the tangent and the bitangent? You should be able to calculate the bitangent from the vertex normal and the tangent, I mean if you always assume the tangent is 'up'...

##### Share on other sites

Yes, you can calculate the bitangent from the normal and the tangent since they form an orthonormal basis. Quite a few engines do that.

##### Share on other sites

Hey guys, you are talking but nothing makes sense atm.

When a Normal Map is created for use in Tangent space, the normal stored in the map is different from this "built in" case. The normal stored in the Tangent Space Normal Map is the offset to apply the normal passed from the vertex shader?

Yes. The tangent space normal map normals are pointing in the correct direction when your matching texture is facing you on your screen. If you wrap that texture around an object, the pixels/normals you pull out of the normal map in the shader will still be in screen space, so you have to apply them to the surface normal at that pixel.

So in the vertex shader, you'll need to pass the vertex normal, the normal map normal and the tangent/bitangent vectors to the pixel shader. The tangent/bitangents vectors coupled with the vertex normal is enough to orient the normal map normal [technical details skipped] to the correct vector. You can then use this in your lighting calculation.

One thing I was wondering actually, if the normals in the normal map always point generally 'out' of screen space, why do you need both the tangent and the bitangent? You should be able to calculate the bitangent from the vertex normal and the tangent, I mean if you always assume the tangent is 'up'...

Wait, so the Vertex Shader samples the Normal Map? That can't be right.

Is this correct:
1. From the vertex shader 3 vertices(with normals) are outputted.
2. In the pixel shader; an interpolated normal (N1) from these 3 vertex normals is passed to the PS.
3. A sampled Normal (N2) from the normal map is retrieved.
4. Using (N1) and (N2) we create a transform matrix (TBN).
5. We then use the (TBN-1) to transform the light vector in world space (LW) to Tangent space (LT).
6. We then use either (N1) or (N2) or a calculation of both to DOT it with (LT).
7. Use result to shade the pixel.

What is wrong here?

##### Share on other sites
An optimization you can make when doing forward rendering is to calculate the TBN matrix in the vertex shader and use it to pre-transform the lightvector into tangent space before sending the transformed lightvector into the pixel shader. Then in the pixel shader, you read the normal from the normal map and perform dot with the incoming light vector. Since the incoming light vector is already pre-transformed into tangent space you don't need to modify it or do anything weird with the normal from the normal map. You'll need to normalize the incoming lightvector in the pixel shader, since interpolation of might cause some shortening.

And no, you don't sample the normal map in the vertex shader.

Regarding your posted list of steps:

1. From the vertex shader 3 vertices(with normals) are outputted.

No. A vertex shader operates only on a single vertex at a time. Values for each vertex given to the shader are calculated and output to an intermediate assembly stage. For a triangle, once the 3 vertices of the triangle have been processed by the shader, then this intermediate stage of the driver calculates interpolated versions of these three sets of data, and hands these interpolated values to the pixel shader as the set of input data for the current pixel.

2. In the pixel shader; an interpolated normal (N1) from these 3 vertex normals is passed to the PS.

Basically, yes.

3. A sampled Normal (N2) from the normal map is retrieved.

Yes. Note that the normal stored in a tangent-space normal map is encoded, so this step will need to include steps to decode the normal. Particularly, the x and y components need to be multiplied by 0.5 then subtract 0.5, to correct them from the [0,1] coordinate space to the [-1,1] coordinate space. The z component, or blue channel, is typically set to 1 so once the x/y components are decoded the whole vector needs to be normalized to unit length. This decoded normal represents the surface normal of the fragment in tangent space.

4. Using (N1) and (N2) we create a transform matrix (TBN).

No, the TBN matrix is constructed from the vertex normal, the tangent and the bi-tangent. These three vectors form a "miniature" 3D coordinate space, if you will, where the normal corresponds to the local Z axis of the space, and the tangent and bitangent correspond to the X and Y axes of the space. These 3 vectors need to be perpendicular (orthogonal) to one another, just as the global X, Y and Z axes are perpendicular to one another. Typically, a tangent vector for each vertex is calculated in a pre-process pass (probably during model export at asset creation, or some sort of process pass either at a later stage or when the model itself is loaded into the game) and passed to the vertex shader as an attribute along with the normal and vertex position. The bitangent can be calculated in the shader by taking the cross-product of the normal and the tangent.

Once the TBN matrix is calculated, it is used to transform the light vector, which is originally in World space. After the transformation, the light vector will now be pointing relative to this miniature coordinate space (called tangent space). Since the decoded normal is also pointing relative to the tangent space, then a dot product between the light vector and the decoded normal vector will calculate the correct shading for the pixel. This shading is then applied to the diffuse color to get the final diffuse value for the fragment.

##### Share on other sites

Ok, lets see if I got it:

1. In the Vertex Shader, the vertex is skinned to bones and transformed, along will its Normal also be transformed from model space to screen space. Since severeal faces can share the same Vertex, that Vertex normal is an avarage of neighbouring face normals. This means that the local  "miniature" 3D coordinate space (TBN) for each vertex is not perpendicular on the faces, but gradually change as you move along the face. The Vertex Normal which is an avarage of neighbouring face normals and transformed to screen space makes up the 'N' component of the TBN matrix.

2. The tangent (T) and bitangent (B) is calculated using the Vertex 'N' component and the Vertex UV coordinate.

3. Still in the Vertex Shader, the Light vector in world space (LW) is transformed with (TBN-1) to a Light vector in tangent space (LT), and then passed to the Pixel shader to be interpolated over the face.

4. Since TBN changes for each vertex, the Light vector (LT) (interpolated before it reaches the Pixel Shader) will change direction as we go along the face for each fragment?

5. In the Pixel Shader the transformed and interpolated Light vector (LT) gets DOTed with the sampled (decoded) Normal from the Normal Map.

6. The result is used to shade the pixel/fragment.

Did I get it right?