Hello.

We're trying to bake normal maps for a low-poly model from a high-poly model in Blender. The output is weird as hell with lots of color gradients that seemingly seem to turn smoothed out normals back into sharp edged objects again. (There are even seemingly inverted normals and other nonsensical crazy normals, but that's not really the problem.) The problem here is that these smooth--->sharp normal maps don't show up correctly in our own engine. It seems like even a tiny minor difference in algorithms is enough to completely destroy the look, as a surface that is supposed to be flat suddenly becomes a tiny bit bent. What I want to know is what exactly I should do to get the exact same result as Blender gets in our game engine.

>What normals, tangents and bitangents are used for the baking?
>Is it possible to use tangent-less shaders that calculate the tangents and bitangents using derivatives (this is what we use right now) and get correct results?

>If not:
- Is it possible to generate the tangents (in an identical way to the way Blender does it), or do they have to be exported by Blender to be correct?

- When are normals, tangents and bitangents supposed to be normalized? Per vertex? Per pixel? Not at all? Should the normal mapping result be normalized? What is the exact set of operations that Blender does to apply a normal map (generated by itself) to an object, and how do I replicate those using shaders?

Our current normal mapping code for calculating the tangent space, for reference: http://pastebin.com/cP92PQVr

Any help would be very appreciated. I've been battling this normal map interpretation mismatch for so long now...

If you just used the default setting you are halfway correct.

First you need to set your bake setting like this image

Here I moved to meshes away to show the poly count. When your baking option is "Selected to Active" as I use here you must overlap the two meshes to bake the normal map.

First select the low mesh, confirm it has a texture assigned and then holding shift select the high poly mesh and bake.

The problem you describe happens when:

You don't divide the low poly model into triangles, before baking.(most common)

Blender bakes the normal map for a quad model, then on import to your game engine the model is converted to triangles, completely changing the normal of the mesh. This causes dents or smoothing of flat faces, as the normal map no longer agrees with the mesh.

You use a 16 bit color depth .png instead of 8 bit.

For some reason this can make Blender bake a inverted normal map, no idea why.

Quick fix use only Targa files.

Your game engine and Blender doesn't share the same normal map axis.

Because openGL uses (X,Y,Z) for normal and DirectX uses (X,-Y,Z) some openGL engines will attempt to do it the directX way to make things simple for Max users.

You can check this by inverting the green channel of the normal map.

When baking your low poly mesh doesn't fully overlap the high poly.

It's common practice that the low poly mesh used for baking is slightly larger than the high poly mesh, this works better with ray tracing.

To do this make a copy of your low poly mesh that is unwrapped, move it over the high poly, go into edit mode, use Alt-S to fatten the model and bake. After bake apply the normal map to the original low poly mesh.

This will help prevent artefacts, most of the time you can use the Bias setting to get the same result. This is just better for objects that haven't been exploded yet.

You didn't set both of the meshes to smooth when baking or importing.

This will cause a problem if the mesh used to bake the normal map is smooth or flat, however the one in your engine isn't smooth or flat.

This depends on you exporter and what you want.

Quick fix, use a other exporter and never use the Directx(.x) exporter as you will need to edit it as a text document to fix.

There is many more reason this could happen, it would be nice if you uploaded a image showing the model in Blender and one showing it in your engine. Also uploading the texture will help.

Thanks a lot for your response, we'll be sure to set up our Blender settings correctly, but the problem I'm talking about comes from very subtle errors in direction, not obvious things like inverted normals or inverted Y coordinates. I need to figure out the exact algorithm that Blender uses for normal mapping to ensure that I use the same normals, tangents, bitangents and normalization steps as they do, or subtle errors will be introduced... Still, thank you a lot for all that information, we'll be sure to take it all into consideration.

There is many more reason this could happen, it would be nice if you uploaded a image showing the model in Blender and one showing it in your engine. Also uploading the texture will help.

We're still trying to learn our ways around Blender, but I will try to do this as soon as I can.

I need to figure out the exact algorithm that Blender uses for normal mapping to ensure that I use the same normals, tangents, bitangents and normalization steps as they do, or subtle errors will be introduced... Still, thank you a lot for all that information, we'll be sure to take it all into consideration

This is the long way, maybe for a programmer this is also the right way?

So why not go download the Blender source, it's easy to understand and all the info you need will be in it?

A other option would be contacting one of the developers, the are a bit busy yet they should know what you need, however I think you will get quicker results by browsing the source.

If the source code is not for you and you are making your own engine, then look into smooth shading.

I am a 3D artist, not a graphics programmer, and in my time I have worked with some shifty engines. Chances are that if you upload a quick model showing how you are doing things I will be able to give you a work around.

Sifting through the entire source code of Blender would be a huge amount of work, especially since it's written in a programming language I'm not particularly experienced with... Otherwise, that would indeed be the "easiest" solution.

Here are the results of a baked normal map:

http://imgur.com/a/nWoOA

This normal map for a simple smooth cube was generated using Substance Painter. We tried to make sure that the mesh was triangulated, but the end result still sucks. Although the mesh gets closer to the right result (y is inverted in this as you can see in the normal map, which looked the most correct), but it's... "wobbly" and uneven even though it should be perfectly flat like the original high-poly model... It's clear that I'm using a different tangent basis from, well, everything else in the entire world it seems.

EDIT: Ahaa! This: http://gamedev.stackexchange.com/questions/128023/how-does-mikktspace-work-for-calculating-the-tangent-space-during-normal-mapping seems to be exactly what I'm looking for!!! Of course it's unanswered...

I don't know what software you are using to bake your normal map, however it is blurring the result that is why you get artifacts.

Are you using some kind of custom normal map software or script?

If you are looking into how to bake normals in general, you should get the openGL orange book. I know it's in there as I relearned some of the shaders a year ago.

I'm pretty sure the blurring is just a different style of filling out the unused space of the normal map. The internals look identical after all. That normal map was baked with Substance Painter.

Here you can see the difference between the two normal maps, this is why the artifacts form.

I'm pretty sure the blurring is just a different style of filling out the unused space of the normal map.

You can see that the blurring does overlap the normal map, not visible to the human eye however the shaders still notice it.

That normal map was baked with Substance Painter.

I don't use Substance myself and have no idea if this is a problem with a setting or technique.

To determine if it is Substance use Blender or Xnormal to bake normal maps, both are free and both uses the standard tangent shader that is used by most game engines. If the normal map from Blender or Xnormal doesn't have a artifact then it's a problem with a setting in Substance.

If you want I can upload .blend that shows how to bake normal maps in Blender.