Hm, I have read up on the whole thing and, having read
this and
this article, I now understand the following:
Quote:
The purpose of transforming a normal by the world matrix is to rotate it so that it's a direction in world-space. This is typically done by only using the 3x3 portion of the world matrix, since this includes all of the rotation data and none of translation (you don't want to translate a normal).
The 3x3 portion of the matrix also can contain scaling data. When it contains scaling data, the normal is scaled by the scaling factor. As long as the scale is uniform (which means that xScale == yScale == zScale), the scaling is okay provided your normalize the resultant normal vector before using it. However if the scale is non-uniform, this won't work. The solution in the case of non-uniform scaling is to calculatate the inverse transpose outside the shader, pass it in, and then transform normals (and also tangents/bitangents, if necessary) by that matrix.
So in summary: if you're using no scaling or uniform scaling in your world matrix you can use the world matrix, if you're using non-uniform scaling then you need to use the inverse transpose.
To be on the safe side, my new plan is to
send the inverse transpose to the shader and multiply my normals with that. This is my new shader code:
float4x4 matWorldViewProj;float4x4 matWorld;float4x4 matWorldIT;float4 lightPosition;float4 lightAmbientColor;float4 lightDiffuseColor;float lightIntensity;float lightCutoff;struct VS_Input{ float4 Position : POSITION; float2 UV : TEXCOORD0; float3 Normal : NORMAL;};struct VS_Output{ float4 Position : POSITION; float2 UV : TEXCOORD0; float4 Normal : TEXCOORD1; float4 PositionWorld : TEXCOORD2;};VS_Output MyVertexShader ( VS_Input In ){ VS_Output Out; // transform vertex into projection space for rendering Out.Position = mul ( In.Position, matWorldViewProj ); // simply pass UV coordinates on Out.UV = In.UV; // pass the normal on Out.Normal.xyz = In.Normal.xyz; Out.Normal.w = 0.0f; // get position in world space Out.PositionWorld = In.Position; return Out;}struct PS_Input{ float2 UV : TEXCOORD0; float4 Normal : TEXCOORD1; float4 Position : TEXCOORD2;};struct PS_Output{ float4 Color : COLOR;};PS_Output MyPixelShader ( PS_Input In, sampler2D tex0 ){ PS_Output Out; In.Normal.w = 0.0f; // normalize normal and multiply it by the // inverse transpose of the world matrix In.Normal = normalize ( mul ( In.Normal, matWorldIT ) ); In.Position.w = 1.0f; // bring position into world space In.Position = mul ( In.Position, matWorld ); // look up the diffuse color from the texture float4 texture_color = tex2D ( tex0, In.UV ); float4 lightPos = lightPosition; lightPos.w = 1.0f; // get the normalized vector from this pixel to light float3 LightVector = normalize ( lightPos - In.Position ); // calculate lighting float3 light = max ( dot ( LightVector, In.Normal ), 0.0f ); // calculate the pixel color (ambient and diffuse) Out.Color.rgb = ( texture_color.rgb * lightAmbientColor.rgb ) + ( texture_color.rgb * lightDiffuseColor.rgb * light ); // support transparency Out.Color.a = texture_color.a; return Out;}
In my application, what I do before rendering a model is I take the desired position, rotation and scale for that model, multiply these together and call "device->SetTransform ()" for the world matrix in order to place it. Then I retrieve the world-, view- and projection matrices and send the world matrix and the world-view-projection matrix to the shader. Then I do this:
D3DXMATRIX worldIT_matrix;device->GetTransform ( D3DTS_WORLD, &worldIT_matrix ); D3DXMatrixInverse ( &worldIT_matrix, NULL, &worldIT_matrix );
...to calculate the inverse transpose of the world matrix and send that to the shader ("matWorldIT").
What this is does it looks almost like the lighting flipped but not entirely. Hard to explain but it's definitely still absolutely wrong.
Any more input/ideas? I don't really need non-uniform scaling at this point, so I also tried just casting the matWorld to (float3x3) for the multiplication with the normal but that doesn't pass my compiler (a post in one of the discussions I linked to earlier said you could).
As you can see in the first shot, the light refuses to brighten up that middle part of the wall there. This dark area moves with the light somewhat (second image). Sometimes it gives the impression that it might follow edges of the level geometry (ie. the faces) but it's kind of hard to tell.
This is my shader:
