Hey guys,

I'm very tired right now from working all day since 8am, so perhaps I'm overlooking something... But I'm having a weird problem with a trivial point light shader. I've gotten my engine rendering its first geometry, and I've just been running some unit tests and making sure everything works properly. Everything has been looking great... I've spent weeks and weeks on end working without seeing a result, and tonight was the first time I've had any payoff! This engine is very big and very complex, so you can imagine how tough its been to work for several weeks without feeling any sense of reward lol...

Anyway, I was just screwing around and implementing some very trivial shaders for fun/testing. But all of a sudden I ran into a problem... For some reason, lighting calculations simply will not work! I've tried tampering and changing things around over and over to no avail. It may just be because I'm tired, but I'm starting to wonder if something is seriously wrong here... Here is the problematic shader:

float4x4 world : World;
float4x4 view : View;
float4x4 proj : Projection;
float3 lightPos;
float lightPow = 10;
float4 vertColor = float4(1,1,1,1);
struct VS_IN {
float4 pos : POSITION;
float3 normal : NORMAL;
};
struct PS_IN {
float4 pos : SV_POSITION;
float3 normal : NORMAL;
};
PS_IN VS( VS_IN input )
{
PS_IN output = (PS_IN)0;

float4x4 _wvp = mul( mul(world, view), proj );

output.pos = mul( input.pos, _wvp );
output.normal = mul( input.normal, _wvp );

return output;
}
float4 PS( PS_IN input ): SV_Target {
float3 ldir = normalize( lightPos - input.pos.xyz );
float lightAmount = dot( input.normal, ldir );

float4 result = (vertColor * lightAmount);
return result;
}
RasterizerState DisableCulling
{
CullMode = NONE;
};
technique10 RenderSolid {
pass P0 {
SetRasterizerState( DisableCulling );

SetGeometryShader( 0 );
SetVertexShader( CompileShader( vs_4_0, VS() ) );
SetPixelShader( CompileShader( ps_4_0, PS() ) );
}
}

I'm rendering a quad, and every time it comes out black... It appears that the result of the dot function is always returning 0, which makes no sense to me... Why? What is going on here?

output.normal = mul( input.normal, _wvp );
You need to transform normal by inversed transposed world matrix. But if it doesn't contain scaling, it'll be equal to float3x3 part, meaning you can just typecast.
If you transform by view and/or projection it has other meaning, and it breaks your point light.

Wow, I'm an idiot lol... It's been a while since I wrote any shaders... What's sad is that I had the solution right in front of my face, open in FX Composer, in some of my old shaders. But it took you saying it before I realized what I did. So thanks a million for saving me some time and not giving me a hard time! :-)

Ok, I feel like an even bigger idiot now because I rewrote the problematic shader and tried to use the proper calculations yet I'm STILL getting totally whacky and off results? WTF is wrong with me? lol... I've never been an expert on HLSL, but I've done trivial things like this a million times in the past. Though it has been a good while since I wrote any shaders... Here is what my new shader looks like:

/*-----------------------------------------------------------------------------*
* GLOBAL VARIABLES ::
*-----------------------------------------------------------------------------*/

float4x4 World : World;
float4x4 View : View;
float4x4 Proj : Projection;
float4x4 WorldITX : WorldInverseTranspose;
/*---------------------------------
* TWEAKABLE PARAMS ::
*---------------------------------*/
float4 VertColor = {1,1,1,1};
float3 LightPos : Position
<
string Object = "PointLight0";
string UIName = "Light Position";
string Space = "World";
> = {-0.5f,2.0f,1.25f};
/*-----------------------------------------------------------------------------*
* TYPE DEFINITIONS ::
*-----------------------------------------------------------------------------*/

/*---------------------------------
* VS INPUT ::
*---------------------------------*/

struct AppData_PN {
float4 Position : POSITION;
float3 Normal : NORMAL;
};
/*---------------------------------
* PS INPUT ::
*---------------------------------*/

struct PSData_PN {
float4 Position : POSITION;
float3 Normal : NORMAL;
};

/*-----------------------------------------------------------------------------*
* VERTEX SHADERS ::
*-----------------------------------------------------------------------------*/
PSData_PN mainVS( AppData_PN input ) {

float4x4 wvp = mul( mul( World, View ), Proj );

PSData_PN output = (PSData_PN)0;

output.Position = mul( input.Position, wvp );
output.Normal = normalize(mul( float4(input.Normal, 1), WorldITX ).xyz);

return output;
}
/*-----------------------------------------------------------------------------*
* PIXEL SHADERS ::
*-----------------------------------------------------------------------------*/
float4 mainPS( PSData_PN input ) : SV_Target {

float3 lightVec = normalize( LightPos - input.Position.xyz );

float lightPow = dot( input.Normal, lightVec );

return (VertColor * lightPow);
}
/*-----------------------------------------------------------------------------*
* TECHNIQUES & RENDER SETUP ::
*-----------------------------------------------------------------------------*/
/*---------------------------------
* RENDER STATES ::
*---------------------------------*/

RasterizerState DisableCulling {
CullMode = NONE;
};

/*---------------------------------
* TECHNIQUES ::
*---------------------------------*/
technique10 pl_PosNorm {
pass p0 {
SetRasterizerState( DisableCulling );
SetGeometryShader( NULL );
SetVertexShader( CompileShader( vs_4_0, mainVS() ) );
SetPixelShader( CompileShader( ps_4_0, mainPS() ) );
}
}
/*-----------------------------------------------------------------------------*
*******************************************************************************
*-----------------------------------------------------------------------------*/

My lighting just comes out plane wrong... I move the point light all around and get nothing... a black quad. To ensure it is not my engine I tried the same thing in FX Composer (which we know for a fact will set the proper geometry, matrices, etc) and I get the same exact result. If I flip/rotate the quad in an odd way where it's almost side-on to the light THEN I get lighting...which is totally wrong. So I know its my sorry little shader...

So what is wrong with my shader? Better yet, what is wrong with ME? lol...

Nevermind... I figured it out! I'm just an idiot!

I was trying to calculate the lighting vector (direction from light to pixel) from the vertex position transformed by the World-View-Projection matrix. I actually should've been calculating this direction in WORLD space! Problem solved! And thanks to Ripiz for reminding me of the proper way to get the transformed normal vector!

EDIT:

Well, not quite "problem solved"... The effect which works fine in FX Composer doesn't work in SlimDX... OMG...

Anyone know why a shader that works fine in FX Composer would not work in D3D10/SlimDX? One thing I noticed is that in your pixel shader's input structure this will not work:

struct PS_IN
{
float4 Position : POSITION; // doesn't work in D3D, only in FXC
};

struct PS_IN
{
float4 Positon : SV_POSITION; // works in D3D
};

I'm thinking it has something to do with the differences between rendering in FX composer vs SlimDX/D3D10... But I can't figure it out... The shader is working 100% perfectly as expected in FXC, but in D3D I'm still getting a black quad. I've checked all the inputs and everything is correct. Once again, it's the freakin shader...

Here is the code which works correctly in FXC, but will not work in D3D10/SlimDX:

/*-----------------------------------------------------------------------------*
* GLOBAL VARIABLES ::
*-----------------------------------------------------------------------------*/

float4x4 World : World;
float4x4 View : View;
float4x4 Proj : Projection;
float4x4 WVP : WorldViewProjection;
float4x4 WorldITX : WorldInverseTranspose;
/*---------------------------------
* TWEAKABLE PARAMS ::
*---------------------------------*/
float4 VertColor = {1,1,1,1};
float3 LightPos : Position
<
string Object = "PointLight0";
string UIName = "Light Position";
string Space = "World";
> = {-0.5f,2.0f,1.25f};
/*-----------------------------------------------------------------------------*
* TYPE DEFINITIONS ::
*-----------------------------------------------------------------------------*/

/*---------------------------------
* VS INPUT ::
*---------------------------------*/

struct AppData_PN {
float4 Position : POSITION;
float3 Normal : NORMAL;
};
/*---------------------------------
* PS INPUT ::
*---------------------------------*/

struct PSData_PN {
float4 Position : SV_POSITION;
float3 Normal : NORMAL;
float4 PositionW : TEXCOORD1;
};

/*-----------------------------------------------------------------------------*
* VERTEX SHADERS ::
*-----------------------------------------------------------------------------*/
PSData_PN mainVS( AppData_PN input ) {

PSData_PN output = (PSData_PN) 0;

output.Position = mul( input.Position, WVP );

output.Normal = normalize( mul( float4(input.Normal, 1), WorldITX ).xyz );
output.PositionW = mul( input.Position, World );

return output;
}
/*-----------------------------------------------------------------------------*
* PIXEL SHADERS ::
*-----------------------------------------------------------------------------*/
float4 mainPS( PSData_PN input ) : SV_Target {

float3 lightVec = normalize( LightPos - input.PositionW.xyz );
float lightPow = dot( input.Normal, lightVec );

return (VertColor * lightPow);
}
/*-----------------------------------------------------------------------------*
* TECHNIQUES & RENDER SETUP ::
*-----------------------------------------------------------------------------*/
/*---------------------------------
* RENDER STATES ::
*---------------------------------*/

RasterizerState DisableCulling {
CullMode = NONE;
};

/*---------------------------------
* TECHNIQUES ::
*---------------------------------*/
technique10 pl_PosNorm {
pass p0 {
SetRasterizerState( DisableCulling );
SetGeometryShader( NULL );
SetVertexShader( CompileShader( vs_4_0, mainVS() ) );
SetPixelShader( CompileShader( ps_4_0, mainPS() ) );
}
}
/*-----------------------------------------------------------------------------*
*******************************************************************************
*-----------------------------------------------------------------------------*/

Ok, I've narrowed it down to this... There's either a problem with how the shader is accepting vertex input, or something is wrong deep in my application.

I found this out because I changed the line where I transform the vertex normal to this:

output.Normal = normalize( mul( float4(0, 0, -1, 1), WorldITX ).xyz );

Then the point light suddenly started working correctly! So that tells me that the normal vector the vertex shader is getting is wrong. Because a hard-coded vector of {0, 0, -1} which is the normal vector of my quad works fine. How could this be happening? I'd seriously hate to have to write an entire SlimDX testing application just to test this shader and see if the problem is the shader or the application...

EDIT:

I've noticed that DirectX10 freaks out if you try to use a float3 as vertex position input, even if you use "float4(Position, 1)" in the shader to make it transform properly with the WVP matrix. Could it be doing the same for the normal vector, which is in float3 form? Could changing it to float4 be a "magic" fix?

BTW, why does D3D flip out over using a float3 for vertex position? When I think about the huge number of vertices a large scene is comprised of that wasted 4-bytes of memory really adds up... It'd be nice to be able to use float3 for position too...

EDIT:

Nope, changing the normal to Vector4/float4 doesnt work either... WTF... This really makes no sense...

Alright, sigh... It's my freakin engine... There's a bug somewhere. I modified the SlimDX minitri example to incorporate my shader into it and the shader actually works fine lol...

The bug hunt is on!

Solved... That was a tough one! FYI it was all because I was incorrectly generating the mesh's input layout. I use a special attribute class which is applied to fields in vertex-types structures so that input elements for meshes can be generated on-the-fly. I had specified the attribute for the vertex Normal field incorrectly. For some odd reason, in my bleary-eyed sleepiness, I mistook the purpose of some of the InputElement fields... For some reason I thought "Slot" was to be treated as an index for the field and that AlignedByteOffset was not from the base address of the vertex structure but from the last field in the structure lol... It took me a little while to figure out what I was doing wrong...however, my debugger saved the day lol. But now everything works correctly and I'm now feeling quite proud of and happy with this engine. Now that it's actually working correctly I can see for the first time just how well it's coming along!