Normal mapping? Tangents? Binormals?

Started by
37 comments, last by es 12 years, 6 months ago

Hey es,

Very interesting. Just to clarify could you
please elaborate on what you mean by 11%?
Also could you show your pixel shader? And also are you using same texture format in both cases?

Has anyone else here tried some different configurations?

Cheers,

Morten.


Hi, Morten!

I used the same BC5 textures in both cases. Here are my shaders:

Vertex Shader(derivative maps)
[source lang="cpp"]
layout(location=0) in vec3 Position;
layout(location=1) in vec2 UV;
layout(location=2) in vec3 Normal;

out block {
vec2 vUV;
vec3 vPos;
vec3 vNormal;
vec2 vScaleDuDv;
flat mat3 mNormalsWV;
} Out;

void main() {
Out.vScaleDuDv=mWorldView[gl_InstanceID][3].xy;
Out.mNormalsWV=mat3(cross(mWorldView[gl_InstanceID][1].xyz,mWorldView[gl_InstanceID][2].xyz),
cross(mWorldView[gl_InstanceID][2].xyz,mWorldView[gl_InstanceID][0].xyz),
cross(mWorldView[gl_InstanceID][0].xyz,mWorldView[gl_InstanceID][1].xyz));
Out.vUV=UV;
Out.vNormal=Normal;
Out.vPos=Position;
gl_Position=vec4((vec4(Position,1.0)*mWorldView[gl_InstanceID]).xyz,1.0)*mProjection;
}
[/source]

Fragment Shader(derivative maps)
[source lang="cpp"]
uniform sampler2D texDiffuse;
uniform sampler2D texBump;

in block {
vec2 vUV;
vec3 vPos;
vec3 vNormal;
vec2 vScaleDuDv;
flat mat3 mNormalsWV;
} In;

layout(location=0) out vec4 FragColor;

//------------------------------------------------------------------------------
// mmikkelsen3d.blogspot.com/2011/07/derivative-maps.html
//------------------------------------------------------------------------------
vec3 GetUnNormalizedBumpNormal(in vec2 dBdUV,in vec2 vUV,in vec3 vPos,in vec3 vNormal) {
// chain rule
vec2 vDxUV=dFdx(vUV);
vec2 vDyUV=dFdy(vUV);
float dBs=dot(dBdUV,vDxUV);
float dBt=dot(dBdUV,vDyUV);
vec3 vSigmaS=dFdx(vPos);
vec3 vSigmaT=dFdy(vPos);
vec3 vR1=cross(vSigmaT,vNormal);
vec3 vR2=cross(vNormal,vSigmaS);
float fDet=dot(vSigmaS,vR1);
vec3 vSurfGrad=sign(fDet)*(dBs*vR1+dBt*vR2);
return (abs(fDet)*vNormal-vSurfGrad);
}

void main() {
vec2 vDuDv=In.vScaleDuDv*(texture(texBump,In.vUV).xy*2.0-1.0);
vec3 vBumpNormal=normalize(GetUnNormalizedBumpNormal(vDuDv,In.vUV,In.vPos,normalize(In.vNormal))*In.mNormalsWV)*(gl_FrontFacing ? 1.0:-1.0);
vec4 vDiffuse=texture(texDiffuse,In.vUV);
FragColor=vec4(ToneMapFilmicU2(vSunColor*(max(dot(vSunDirection,vBumpNormal),0.0)+0.01)*vDiffuse.rgb),vDiffuse.a);
}
[/source]

Vertex Shader(tangent space)
[source lang="cpp"]
layout(location=0) in vec3 vPosition;
layout(location=1) in vec2 vUV;
layout(location=2) in vec3 vNormal;
layout(location=3) in vec4 vTangent;

out block {
vec3 vTangent;
vec3 vBiTangent;
vec3 vNormal;
vec2 vUV;
} Out;

void main() {
mat3 mNormalsWV=mat3(cross(mWorldView[gl_InstanceID][1].xyz,mWorldView[gl_InstanceID][2].xyz),
cross(mWorldView[gl_InstanceID][2].xyz,mWorldView[gl_InstanceID][0].xyz),
cross(mWorldView[gl_InstanceID][0].xyz,mWorldView[gl_InstanceID][1].xyz));
Out.vUV=vUV;
Out.vNormal=normalize(vNormal*mNormalsWV);
Out.vTangent=normalize(vTangent.xyz*mNormalsWV);
Out.vBiTangent=normalize(vTangent.w*cross(Out.vNormal,Out.vTangent));
gl_Position=vec4((vec4(vPosition,1.0)*mWorldView[gl_InstanceID]).xyz,1.0)*mProjection;
}
[/source]

Fragment Shader(tangent space)
[source lang="cpp"]
uniform sampler2D texDiffuse;
uniform sampler2D texBump;

in block {
vec3 vTangent;
vec3 vBiTangent;
vec3 vNormal;
vec2 vUV;
} In;

layout(location=0) out vec4 FragColor;

void main() {
vec3 vBump;
vBump.xy=2.0*texture(texBump,In.vUV).xy-1.0;
vBump.z=sqrt(1.0-dot(vBump.xy,vBump.xy));
vec4 vDiffuse=texture(texDiffuse,In.vUV);
vec3 vBumpNormal=normalize(normalize(In.vTangent)*vBump.x+normalize(In.vBiTangent)*vBump.y+normalize(In.vNormal)*vBump.z)*(gl_FrontFacing ? 1.0:-1.0);
FragColor=vec4(ToneMapFilmicU2(vSunColor*(max(dot(vSunDirection,vBumpNormal),0.0)+0.01)*vDiffuse.rgb),vDiffuse.a);
}
[/source]

And here are screenshots with test scene:

Sponza (derivative maps approach, 373 fps)
http://imageshack.us...7/testdudv.jpg/

Sponza (tangent space approach, 419 fps)
http://imageshack.us.../6/testtbn.jpg/

Well, 373.0 fps / 419.0 fps ~ 0.89 -> derivative maps approach is slower than tangent space approach at 11% (in my tests)

Dima.
I am the God of War! None shall defy me!
Advertisement

[quote name='Ftn' timestamp='1314968619' post='4856709']
[quote name='ProfL' timestamp='1314803962' post='4855914']
while I'm hijacking this thread anyway, 2nd thing that confuses me, is it really valid to average the tangents matrices on vertices? they usually represent individual spaces for one triangle(-edge) a neighboring triangle might have a completely unrelated space which is even flipped due to UV mirroring or something. I think if the orthogonalization would remove anything but rotation, this could work if you propagate per triangle the flipping bit, but that leads to my first question.

Seams in UV mapping, or hard edges for normal smoothing, causes split vertices. In those cases you can't average the tangents for the edge, and the tangent spaces for the neighboring triangles are not continuous.
[/quote]that's kind of obvious :D

I'm talking of course about shared edges/vertices. without orthogonalization you clearly don't have to average the normal, but the tangent and bitangent kinda would need that for smoothing.




@Digitalfragment thank you for confirming my worries, maybe any more takers with opinion? I wonder how you guys are dealing with those issues, obviously no game stores a full tangent matrix per vertex, all seem to run with orthogonalization mostly because the inverse in shader is just a simple transpose, not an expensive full 3x3 inverse (although it could be normalized which would at least save calculating the determinant and the division, I assume).


[/quote]


Some of your questions are discussed here

--> http://wiki.blender.org/index.php/Dev:Shading/Tangent_Space_Normal_Maps

and the rest are thoroughly discussed in the thesis referenced there.
It discusses things like when to average and when to split and also which approximations
are used in games, and why, and how the baker should support these choices to avoid visual errors (not the other way around).






Hi, Morten!


Well, 373.0 fps / 419.0 fps ~ 0.89 -> derivative maps approach is slower than tangent space approach at 11% (in my tests)

Dima.



Thanks a lot for sharing.

I have a few comments. One observation is that you're applying a transformation in the pixel shader
which shouldn't be there. You should simply be passing in the normal and surface position in the same space to
the interpolators such that no additional transformations are necessary.
In other words take out vScaleDuDv and mNormalsWV entirely. You're needlessly burning up
interpolators. Using fewer interpolators is supposed to be one of the advantages.

Another observation is that I neglected to mention in my paper that you need a bump scale when doing bump mapping.
So you should have a user-defined scalar constant that you scale dBs and dBt by before using them in the perturbation.
For derivative maps you can also apply this scale to dBduv if you want (the sampled derivative).

That being said even if you change this it is quite possible it'll still be a little bit slower on current gen.
However, I believe this method will be faster on coming generations of cards since it replaces interpolators with a little bit of math.

However, even if I am wrong, you have to realize what you get with this method in return. You don't need to store any tangent spaces
in memory or split any vertices. You have the option of using BC4 height maps instead of derivative maps if you're starving for memory.
Finally, it allows you to, easily, bump map synthesized surfaces such as subdiv or even surfaces being subjected to "unconventional deformations (not just skinning)" such as cloth or water (ocean/lakes) without having to recompute tangents at the vertices.

For subdiv alone this method is really extremely convenient compared to traditional normal map baking.
It's not a very good idea to bake using the control mesh as your low poly when you intend to apply it
to a subdiv surface. And it doesn't entirely make sense to use traditional tangent space generation at the vertices of the
full res subdiv surface either. And even if you did you wouldn't be able to, efficiently, generate the same spaces on the gpu.

The other thing I want to point out is your pixel shader is very short. If you compare it to a next gen shader
with possibly fancier lighting models, shadows, rims, high lights etc then the difference in perturbing a normal one way or the other
becomes very marginal relative to the full fps.

Anyway, glad you shared all your results and took the time to give it a shot.

Morten.

In other words take out vScaleDuDv and mNormalsWV entirely. You're needlessly burning up
interpolators. Using fewer interpolators is supposed to be one of the advantages.



Morten, thanks for the tips!
I would like to say few words about my variables (but my english is not so good and perhaps i didn't understand your words properly). The vScaleDuDv is a user-scale for dBs and dBt and the mNormalsWV is a matrix which apply world-view transformations to normals. The world matrix can has scale factor and there are 2 options for computation of bumped-normal:
1. I can get bumped-normal in object space and then transform it in camera space(there is matrix multiplication in pixel shader).
2. I can get bumped-normal directly in camera space(and all matrix multiplications are made in vertex shader)

These 2 approaches give different results and i had doubts about right way of computation. In the end i chose the first approach.

Naughty Dog rules!
Uncharted rules!

Dima.
I am the God of War! None shall defy me!



Morten, thanks for the tips!
I would like to say few words about my variables (but my english is not so good and perhaps i didn't understand your words properly). The vScaleDuDv is a user-scale for dBs and dBt and the mNormalsWV is a matrix which apply world-view transformations to normals. The world matrix can has scale factor and there are 2 options for computation of bumped-normal:
1. I can get bumped-normal in object space and then transform it in camera space(there is matrix multiplication in pixel shader).
2. I can get bumped-normal directly in camera space(and all matrix multiplications are made in vertex shader)

These 2 approaches give different results and i had doubts about right way of computation. In the end i chose the first approach.

Naughty Dog rules!
Uncharted rules!

Dima.


Thanks :)

As for the shader I am still having a tough time wrapping my brain around your vScaleDuDv. It looks like you are writing the .xy
of the translation part of your world view to it? The bump scale I had in mind is a single value (1 float) specified either per material, per object or both.

As for the transformation part you are right that if your aim is to perturb in object space while doing lighting in view space then you'd be
stuck doing a transformation in your pixel shader. Furthermore, using instancing as it seems you do you'd end up
passing it to interpolators instead of the matrix being a shader constant.

I would definitely do the perturbation in view-space especially when doing instancing to avoid passing a matrix to the interpolators.

It looks like you are writing the .xy of the translation part of your world view to it?

no, my matrices have row-major order(like in DirectX), fourth column of world-view matrix is just float4(0,0,0,1) by default and is not important in computations. So i can use this space for my needs.


As for the shader I am still having a tough time wrapping my brain around your vScaleDuDv. The bump scale I had in mind is a single value (1 float) specified either per material, per object or both.


Well, these lines of code are from your blog:

int2 dim; dbmap.GetDimensions(dim.x, dim.y);
const float2 dBdUV = dim*(2*dbmap.Sample(sampler, In.texST).xy-1);

The vScaleDuDv has the following meaning :
const float2 vScaleDuDv=fUserScale*dim;

And In my case the vScaleDuDv is specified per object (i.e. mWorldView[gl_InstanceID1][3].xy == mWorldView[gl_InstanceID2][3].xy).

Dima.
I am the God of War! None shall defy me!



Well, these lines of code are from your blog:

int2 dim; dbmap.GetDimensions(dim.x, dim.y);
const float2 dBdUV = dim*(2*dbmap.Sample(sampler, In.texST).xy-1);

The vScaleDuDv has the following meaning :
const float2 vScaleDuDv=fUserScale*dim;

And In my case the vScaleDuDv is specified per object (i.e. mWorldView[gl_InstanceID1][3].xy == mWorldView[gl_InstanceID2][3].xy).




Ah ok! You trickster :)

Why don't you simply use getTextureSize() which is what I think GetDimensions() is called in glsl?
Did you check if there is a performance difference when perturbing in view space?

Cheers,

Morten.

[quote name='ProfL' timestamp='1314974241' post='4856757']
[quote name='Ftn' timestamp='1314968619' post='4856709']
[quote name='ProfL' timestamp='1314803962' post='4855914']
while I'm hijacking this thread anyway, 2nd thing that confuses me, is it really valid to average the tangents matrices on vertices? they usually represent individual spaces for one triangle(-edge) a neighboring triangle might have a completely unrelated space which is even flipped due to UV mirroring or something. I think if the orthogonalization would remove anything but rotation, this could work if you propagate per triangle the flipping bit, but that leads to my first question.

Seams in UV mapping, or hard edges for normal smoothing, causes split vertices. In those cases you can't average the tangents for the edge, and the tangent spaces for the neighboring triangles are not continuous.
[/quote]that's kind of obvious :D

I'm talking of course about shared edges/vertices. without orthogonalization you clearly don't have to average the normal, but the tangent and bitangent kinda would need that for smoothing.




@Digitalfragment thank you for confirming my worries, maybe any more takers with opinion? I wonder how you guys are dealing with those issues, obviously no game stores a full tangent matrix per vertex, all seem to run with orthogonalization mostly because the inverse in shader is just a simple transpose, not an expensive full 3x3 inverse (although it could be normalized which would at least save calculating the determinant and the division, I assume).


[/quote]


Some of your questions are discussed here

--> http://wiki.blender....ace_Normal_Maps

and the rest are thoroughly discussed in the thesis referenced there.
It discusses things like when to average and when to split and also which approximations
are used in games, and why, and how the baker should support these choices to avoid visual errors (not the other way around).
[/quote]This text is a bit weird. it claims the tangent space is not asset independent, which implies every mesh, or rather, every triangle in every mesh needs a specifically made texture(-area). This makes the whole point of tangentspace useless, you can just as good use an object space texture and you'd save all the tangent space calculations and extra storage (except for maybe skinned objects).


tangent space is usually used to have the possibility to swap and share textures across arbitrary meshes, quite often those are created by artist using heightmap -> normalmap converter, and if you use the full 3x3 matrix, it's working.

(sorry, I've not read the paper yet, as I'm at work, will read it during lunch, but the Blender text seems to sum that up and sounds not really correct).





Why don't you simply use getTextureSize() which is what I think GetDimensions() is called in glsl?


I was afraid that it could have been expensive :)


Did you check if there is a performance difference when perturbing in view space?


Yeah, I've installed new experimental Nvidia's drivers with OpenGL 4.2 support and here are my results:

derivative maps approach(camera space) - 415 fps
derivative maps approach(object space) - 385 fps
tangent space approach - 425 fps

But, Morten, is this correct to compute perturbing normal in view space when you have any scale factor in world matrix?
Is there any rule or maybe hint how to control user-scale for dBs and dBt or designers specify it "by sight" ?

Thanks for support,
Dima.
I am the God of War! None shall defy me!

This text is a bit weird. it claims the tangent space is not asset independent, which implies every mesh, or rather, every triangle in every mesh needs a specifically made texture(-area). This makes the whole point of tangentspace useless, you can just as good use an object space texture and you'd save all the tangent space calculations and extra storage (except for maybe skinned objects).


tangent space is usually used to have the possibility to swap and share textures across arbitrary meshes, quite often those are created by artist using heightmap -> normalmap converter, and if you use the full 3x3 matrix, it's working.

(sorry, I've not read the paper yet, as I'm at work, will read it during lunch, but the Blender text seems to sum that up and sounds not really correct).



It's complicated. The text is distinguishing between normal fields sampled from a hi-res surface to a lo-res surface
and regular height maps converted into normal maps which is probably what you're thinking of

Imagine for instance a sphere having its normals smoothly perturbed all over using some low frequency noise function.
So the field of normals is all over continuous and smooth. Next let's assume this sphere is represented by at least two islands
in the texture map. Then the text points out if the baker transforms the normal field into tangent space using one transformation
and the pixel shader uses another then you get unwanted hard edges in the lighting during rendering which is entirely true.
In this use-case the texture is not really reusable on arbitrary shapes which is indicative of the issues involved when character artists bake normal maps from sculpts.

In a sense you are correct that they could use object space normal maps when these maps aren't reusable anyway but the part you're missing
is that by transforming into a tangent space you get to reduce the normal map to two components instead of 3. It survives compression better
because some amount of curvature is taken out of the normal field and ultimately allows us to get good results with BC5 and also mirroring is possible.
Furthermore, it allows us to always use the same pixel shader whether the normal map was created using sampling from hi-res to lo-res or by converting a height map into a normal map.

This topic is closed to new replies.

Advertisement