I'm trying to implement lighting in my engine but running into some issue. I'm using Frank Luna's book Beginner's Guide to Programming with DirectX 11, but I'm not sure that I'm reading the descriptions right.

This is the pseudocode that I wrote based on the book's guidelines.

1) Calculate vertex normals for the mesh (on the cpu)

2) Find angle between light direction and vertex normal (in the pixel shader)

               angle = max( dotproduct(light_dir, vertex_normal), 0)

3) Find the color of the pixel if the texture was applied to it

              textured_pixel_color = sample(texture)

4) Combine the light color with the textured pixel color 

              final_color = lightColor * textured_pixel_color

5) Combine the intensity with the final_color to get the final_pixel_color_and_intensity

                final_pixel_color_and_intensity = angle * final_color

This is it implemented in the pixel shader

cbuffer DirectionalLight : register (b2)
{
	float3 LightDirection;
	float3 LightColor;
};


struct TexturedLitVertex
{
	float4 Pos :   SV_POSITION;
	float3 Normal: NORMAL;
	float2 Uv:     TEXCOORD;
};

float4 PS_DirectionalLight(TexturedLitVertex psInput) : SV_Target
{
	float4 colorAfterTexture = txDiffuse.Sample(triLinearSampler, psInput.Uv);
	
	float lightNormalAngle = max(dot(LightDirection, psInput.Normal), 0.0f);

	return lightNormalAngle * (colorAfterTexture*float4(LightColor, 1.0f));
}

This is the result I'm getting (see image).  What could be the problem?

5 hours ago, JohnnyCode said:

What are the DirectionalLight values u pass as a uniform?

Also you should normalize

psInput.Normal

as it is a per pixel lightning and the normal interpolates.

The light direction is (0.57, -0.57, 0.57) the light color is (1.0f, 1.0f, 1.0f, 1.0f)

The vertex normal is normalized in the vertex shader. 

What do you mean this is a per pixel lighting? I'm trying to do per vertex lighting.

I ended up just using the version in the DirectX samples (June 2010) Tutorial 06: Lighting the pixel shader is much more simplified but it works:

cbuffer DirectionalLight : register (b2)
{
	float3 LightDirection;
	float4 LightColor;
};


struct TexturedLitVertex
{
	float4 Pos :   SV_POSITION;
	float3 Normal: NORMAL;
	float2 Uv:     TEXCOORD;
};


TexturedLitVertex VS_DirectionalLight(float4 Pos : POSITION, float3 Normal : NORMAL, float2 Uv : TEXCOORD0)
{
	TexturedLitVertex vsOut = (TexturedLitVertex)0;

	vsOut.Pos = mul(Pos, World);
	vsOut.Pos = mul(vsOut.Pos, View);
	vsOut.Pos = mul(vsOut.Pos, Projection);

	vsOut.Normal = mul(Normal, (float3x3)World);
	vsOut.Uv = Uv;
	
	return vsOut;
}

float4 PS_DirectionalLight(TexturedLitVertex psInput) : SV_Target
{
	psInput.Normal = normalize(psInput.Normal);

	float4 finalColor = 0;

	//do NdotL lighting
	float4 lcolor = { 1.0f, 1.0f, 1.0f, 1.0f };
	finalColor += saturate(dot((float3)LightDirection, psInput.Normal) * lcolor * txDiffuse.Sample(triLinearSampler, psInput.Uv));
	
	finalColor.a = 1;
	return finalColor;
	
}

This is the working image of it:

7 hours ago, Green_Baron said:

I do all shading in the fragment shader (OpenGL term for pixel shader, i think) and remember having had similar effects.

 

This may be a stupid hint, but may it be that you are lighting the scene from below ? What happens when you change sign of Light direction in:

float lightNormalAngle = max(dot(LightDirection, psInput.Normal), 0.0f);

or change the light position to be positive on all axes ?

 

Just a random thought ...

 Yes, I think I was lighting it from the bottom. I'm not sure it was the only issue though. The light direction values the DirectX sample had were opposite of mine (mine were (0.577, -0.577, 0.577) the sample had ({ -0.577f, 0.577f, -0.577f }) so it might have very well been the issue, but it works now as far as i can tell. 

5 hours ago, JohnnyCode said:

Since you perform taxture sampling in the function I take it for granted you perform the dot product between light direction and normal in pixel function as well, and that is why you need to normalize it in pixel function again, normalizing in vertex function is not sufficient, since as I said, it interpolates.

 

 I added the normalization as you suggested (see code in my previous post), but I'm not sure I understand the reason for it. I'm not changing the value of the normal in the pixel shader. The only time I use it is in the dot product with the light, but why would that change it's value? Anyways, thanks, it works now.

1 hour ago, JohnnyCode said:

Pixel function inputs are streamed from vertex function and are interpolated (wheather a number or a multiple dimension vector) across the rasterized triangle, based on baricentric cordinates of pixel inside the triangle between the verticies. And interpolation does not preserve the length of multidimensional vector, thus to recieve a correct interpretation of dot product, you need to normalize it.

That's interesting, I just didn't know that something was done to the vectors in between the vertex and pixel function. It makes sense now, thanks!