In my engine I have some normal mapped assets. One of them is a barrel with rings that are created through normal mapping. Ever since I had them, sometimes it seems like the rings are not being raised, but instead are lowered into the surface. But only from some angels. I was never sure if it is an illusion or an error.

Now I created a debug view that outputs normals and will be used for some effect. Here are a few screenshots with the barrel, showing places where the direction of the normals seems to change:

[attachment=17402:02_16.png]

[attachment=17403:02_17.png]

[attachment=17404:02_18.png]

How does that look? Is it wrong? The last image, near the cross-hair, slightly to the left, clearly diplays some shenanigans! I also tried changing two channels around, the ones that use the tangent and binormal:

[attachment=17405:02_19.png]

[attachment=17406:02_20.png]

Its hard to say without knowing how the normal maps are generated, what coordinate system you do your lighting and how you are visualizing the normals in your debug shots...are the normal being displayed in tangent space, objects space, world space, screen space? It looks like you are simply displaying the normals as they appear in tangent space; that's fine but it doesn't provide enough information to properly diagnose where things are going wrong. At the very least you need shots that compare normals that you know are correct with normals that you suspect are wrong. This might help:

Tangent Space Derivation

Or at least clue you into some of the subtitles involved. In general per-pixel normals are stored in tangent space (texture space) and transformed into world space before being used. Making sure all your data is in the right coordinate system can be tricky.

Well, all these spaces are confusing and I can't say that I 100% understand them. I am rendering in such a way that the objects rendered with only vertex normals and the same objects rendered with with per pixel normals look identical, except for the added detail.

So I'm guessing that displaying the normals is done in both cases in tangent space? For the vertex normals, I am taking the normals as they appear in the mesh information (God knows what space they are in), basically a vector3 and multiply it with the world inverse transposed matrix. This is the value that gets passed from the vertex to the pixel shader. For the final display, I adjust the value because the vector3 needs to be converted to a color output that is clipped to [0, 1], so I output (value + 1) / 2:

[attachment=17412:02_14.png]

For per pixel normals, things are more complicated. I have the normal map and I need to convert it to tangent space? The conversion is similar to the conversion I do for the bump mapping shaders, so in the vertex shader I multiply the vertex normal and tangent by the world inverse transposed matrix, compute the binormal and send these three values to the pixel shader, where I finally compute the per pixel normal:

float3 normalFromMap = tex2D(normalMap, input.tc).xyz * 2 - 1;
	
float3 n = normalize(float3(input.n * normalFromMap.z + normalFromMap.x * input.t - normalFromMap.y * input.b)); 

Again, for output purposes, I use (n + 1) / 2 and this is the result:

[attachment=17413:02_15.png]

I have also attached the normal map I use for the barrel (as a JPEG because it was too big to upload):

[attachment=17414:02_21.jpg]

The problem is that I'm self though on the subject of shaders and 3D engines, so I wrote and put together most of my shaders and I also created the normal map, so I have no real reference of a correct mesh and its normal mapping. Maybe some cubes...

Any vectors computed from mesh data will invariably be in object space, this includes your normals and tangents. From the code and the screens it looks like you are doing everything correct. The best way to verify is to compare a box without a normal map with a box with a special normal map that has all the normals pointing straight up. If they look the same you can be reasonable sure things are working. Make sure you rotate the cubes around to confirm the tangent space vectors are constructed and treated properly. After that you can add pyramid shaped 'dents' to your normals maps and investigate further. You'll need to get used to mentally converting colors to directions. It helps to draw a coordinate system in front of the camera with axis colors that correspond to the same color those vector would be in a normal map. By the looks of your normal map full red is +x-axis, full green is +y-axis, full blue is +z-axis. This is not how all normal maps are formatted.

These kinds of problems are tricky because a mistake in one place can be canceled out by a mistake someplace else. For example, if the binormal is computed by crossing the tangent and normal in the wrong order this error can be countered by a minus sign instead of a plus sign when summing the projection of the normal onto the tangent space vectors.

One thing you shouldn't do is try to debug this by moving lights around and observing the changing diffuse/spec term. This just complicates things as bugs in your tangent space / normal map treatment may be compounded or undone by bugs in your lighting code. You are on the right track by visualizing the normals directly. Move on to verifying the lighting once you are reasonably sure your normals are being treated properly.

Thank you very much for taking a look! I agree, normal computation seems to be correct.

So I started searching deeper, and I output the tangent and binormals, as seen in attachments. The line of discontinuity that is visible lines up just perfectly with the place where the normals switch direction. It is also the place where the UV mapping of the model has a seam.

I googled it quite a bit and it seems this is a very common problem that doesn't show up in the 3D modelling software because it is built to fix this, but it shows up once you export the mesh. The only partial solution I found that shows some promise and which works on relatively cylindrical wrappings is to subdivide vertically the last quad strip, preferably in the middle, and do you seam there. This way the middle new seam will have tha same angle to the old quad vertices.

Or maybe I can fix the tangents in code somehow. I'll continue and investigate.

Oh yeah, getting closer! I decided that I'll ignore the tangents from the mesh and recompute them myself. I think I am computing them wrong, because if I were to compute them right my point light attenuation bug would be probably fixed or at least improved, but I am getting a lot better results.

Before (directional light with spherical harmonics with n dot l compensation):

[attachment=17458:02_24.jpg]

After:

[attachment=17454:02_25.png]

Wow! Normal mapping looks completely different! The thing is, I tried the same mesh in multiple engines and there were always weird things going on near the rings. Could it be that for so many months I had wrong tangents on my meshes? I need to find who is to be faulted for the wrong tangent. Is it the mesh or the import process.

Here is a closeup of the normals:

[attachment=17456:02_27.png]

[attachment=17457:02_28.png]

Does that look better or correct? Shouldn't the green underside of the ring be swapped with the dark blue one?

There are still a few quirks to fix, like this nasty seam:

[attachment=17455:02_26.png]

Problem pretty much solved!

If you google it, it seems that this is an incredibly widespread problem. I found dozens of forum thread related to normal mapping seams where there is a break in the UV mapping, but zero solutions. Most threads did not manage to figure out the real problem: in the 3D modeling program, vertices in a smooth group have continuous tangents. When exporting, most formats won't store the tangents and the continuous vertices where the UV mapping discontinuity happens will be exported as two vertices, each with a separate UV coordinate. It doesn't matter that the two vertices have the same normals (but it sure does helps that most exporters will handle the normals correctly) because once you compute the tangents, the discontinuity in the UV mapping causes discontinuity in the tangent space.

I tried the fix of breaking a polygon strip in two and setting the seam in the middle, but I'm not 3D modeler and this becomes very hard with complex UV unwraps. But if you can pull it off it works.

So I decided to fix this procedurally after the mesh load. First I wrote a debug mode to detect the seam:

[attachment=17459:03_14.png]

Then I fixed the tangents on the detected seam. The fix is not mathematically sound, but it is a fair approximation for now and it greatly reduced the normal mapping seams:

[attachment=17460:03_15.png]

There is still a subtle seam, but this one is considerably less disturbing. I suspected that this is not due to the new fixed tangents, which look pretty smooth:

[attachment=17461:03_16.png]

So it must be the actual texture of the object that has a seam. First I investigate just the normal mapping effect and looks fine and almost seamless:

[attachment=17462:03_17.png]

The small seam might be because of the fix or because the normal map is not perfectly seamless. And finally I investigate just the texture, without normal mapping and notice the seam that is near identical to the one I was searching for:

[attachment=17463:03_18.png]

Fantastic! So theoretically if the edges of the UV map will get blended and mirrored in an image editor, the seam should be fixed.

I still need to try out my fix for more complex meshes.

So my question to you is as following: how did you fix this? Being such a common problem, found from Unity to UDK imports, I'm probably not the first who has encountered it. What fix did you use? Or do you use some 3D exporters that don't have these problems? If so, could you point them out to me? Blender is really not that consistent with its export tools.

Indeed, I suspect many commercial games have shipped with a spattering of models that have wonky tangent and binormal vectors. As you indicated, this is probably at lest part of your problem. In general exporters seem bad at getting this right in all cases.

The problem with UV seams is widespread and rarely solved properly; however, the cause is fairly simply. The straight forward way to compute normal and tangent vector is on a per triangle basis, when a vertex is shared by multiple triangles the vectors are simply weighted by the area of each triangle and averaged together. This works perfectly until you have a normal-map that is wrapped all the way around a smooth object and one side of the map meets the opposite side of the map on a sequences of edges in the mesh. The UV coordinates are disjoint along this edge so the vertices get duplicated by most exporters (even though they have the same position). When you go to compute your normal vectors at the vertices along this edge they don't get averaged by the triangles on both sides of the edge. While you do want your tangents to be computed without averaging you need the normals to be averaged. One copy of the vertex will incorrectly get the normal from the triangle on the left and the other vertex will gets the normal from the triangle on the right. The way you fix this is to intelligently look at the smoothing groups in the model data and average the normals together along UV seams while leaving the UV coordinates and tangents alone. In the case of the barrel you would identify the sets of vertices that share the same position and smoothing group and average their normals together while leaving the tangents and UVs alone.

You can also get seems where the UVs, tangents and positions along an edge are all continuous but the direction of the binormal gets inverted. This happens when a texture is mirrored across a smoothing group in a 3D modeling package. You can solve the problem in the same way, just average the normals.

Getting all this correct usually means writing your own exporter that converts raw model data from Maya or 3D Studio Max or whatever into a format native to your game.

Indeed, I suspect many commercial games have shipped with a spattering of models that have wonky tangent and binormal vectors. As you indicated, this is probably at lest part of your problem. In general exporters seem bad at getting this right in all cases.

The problem with UV seams is widespread and rarely solved properly; however, the cause is fairly simply. The straight forward way to compute normal and tangent vector is on a per triangle basis, when a vertex is shared by multiple triangles the vectors are simply weighted by the area of each triangle and averaged together. This works perfectly until you have a normal-map that is wrapped all the way around a smooth object and one side of the map meets the opposite side of the map on a sequences of edges in the mesh. The UV coordinates are disjoint along this edge so the vertices get duplicated by most exporters (even though they have the same position). When you go to compute your normal vectors at the vertices along this edge they don't get averaged by the triangles on both sides of the edge. While you do want your tangents to be computed without averaging you need the normals to be averaged. One copy of the vertex will incorrectly get the normal from the triangle on the left and the other vertex will gets the normal from the triangle on the right. The way you fix this is to intelligently look at the smoothing groups in the model data and average the normals together along UV seams while leaving the UV coordinates and tangents alone. In the case of the barrel you would identify the sets of vertices that share the same position and smoothing group and average their normals together while leaving the tangents and UVs alone.

You can also get seems where the UVs, tangents and positions along an edge are all continuous but the direction of the binormal gets inverted. This happens when a texture is mirrored across a smoothing group in a 3D modeling package. You can solve the problem in the same way, just average the normals.

Getting all this correct usually means writing your own exporter that converts raw model data from Maya or 3D Studio Max or whatever into a format native to your game.

Forgot about a subtlety. Sometimes the tangent vectors along a seam will need a nudge such that they become collinear (but still opposite directions). Usually the tessellation takes care of this but not always. Same deal with binormals when dealing with mirrored UVs.