Hi,

I've been trying to use the VFACE semantic to get different shading on the front and back side of my geometry. According to the D3D documentation, VFACE should be positive for frontfacing triangles and negative for backfacing triangles (I guess it would be more concise to say counterclockwise and clockwise?). I got it working in most scenarios. Only when textures are involved, it seems to fail (more precisely: I've got a batch of tests, and only the ones with textures are failing. What happens is that only one side of the geometry gets shaded, or both sides get shaded with the same material). I've been trying to pinpoint the problem, but without succes. In the case where it works, I get shader assembly code like this:

    ps_3_0
          def c0, 0, 1, 0, 0
          dcl vFace
   0:     cmp oC0.xz, vFace, c0.xyyw, c0.yyxw
   1:     mov oC0.yw, c0.xxzy

I don't know the instructions well, but this seems logical: in line 0 something different happens, based on the value of vFace. When I now take one of my shaders that use textures, I get this:

    ps_3_0
          def c8, -1, 1, 0, 0
          dcl_normal v0.xyz
          dcl_texcoord v1.xy
          dcl vFace
          dcl_2d s2
          dcl_2d s3
   0:     cmp r0.x, vFace, c8.x, c8.x
   1:     if_lt r0.x, c8.x
  //blah
  24:     else
  //blah
  45:     endif

That seems very wrong: c8.x gets stuffed in r0.x, no matter what value vFace has! In both cases, the hlsl code is something like:

    void main(in PS_INPUT Input, in float isBackFacing : VFACE, out PS_OUTPUT Output)
      {
      Output.color = float4(0,0,0,1);
      if (isBackFacing < 0)
        //blah
      else
        //blah
      }

Am I misunderstanding this stuff? Are there things I should know about the interaction between VFACE and texturing? I really have no clue why this doesn't work, so any help is much appreciated

BTW: I check all calls into D3D, and have debug output at the highest level, but no errors are reported.

Ok, so it seems this might not be my mistake after all then :-) I currently use the D3DX functionality from the June 2010 SDK. For now I'm still stuck to Windows XP, so I guess switching to newer versions of the compiler won't work (I'm doing runtime shader generation, so I can't compile offline, and the D3DCompiler dlls are only available for Vista and higher). I'll already try whether the issue is fixed on my Windows 7 machine if I use D3DCompile and D3DReflect instead (the documentation is not really clear on whether this will all work with D3D9, but let's find out :-) ).

Are there possibilities to get the June 2010 SDK compiler to do the right thing? Using different statements, or reordering code or somesuch?

If all else fails, I can always return to the flawed approach of using the (possibly interpolated) vertex normals to decide on front vs back.

So I've tried switching to compiler 4.6 (this is the one that comes with VS2012). Before it was 4.3. The disassembled code indicates the version number used to be 9.29.952.3111, while the newer one has version number 9.30.9200.20546. I also tried reducing the optimization level.

Didn't make any difference. Here's the pixel shader's full hlsl code:

struct PS_INPUT
  {
  float3 normal               : NORMAL0;
  float2 textureCoordinate    : TEXCOORD0;
  };

struct PS_OUTPUT
  {
  float4 color        : COLOR0;
  };

void HandleClipping(PS_INPUT Input)
  {
  }


float3 LightDirection0;
float4 LightColor0;

float4 FrontDiffuse;
float4 FrontSpecular;
float FrontShininess;
sampler2D FrontTexture: register(s2);

float4 BackDiffuse;
float4 BackSpecular;
float BackShininess;
sampler2D BackTexture: register(s3);

float LambertFactor(float3 normal, float3 lightDir)
  {
  return saturate(dot(normal, lightDir));
  }

float PhongFactor(float3 normal, float3 lightDir, float shininess)
  {
  float3 H = normalize(lightDir + float3(0,0,1));
  float NdotH = saturate(dot(normal, H));
  return pow(NdotH, shininess);
  }


float4 ShadeFragment(PS_INPUT Input, float isBackFacing)
  {
  if (isBackFacing < 0)
    {
    Input.normal = normalize(Input.normal);
    float4 color = float4(0,0,0,1);
    
    float4 textureColor = tex2D(FrontTexture, Input.textureCoordinate);
    
    float4 diffuse = float4(1,1,1,1);
    diffuse *= textureColor;
    diffuse *= FrontDiffuse;
    
    float4 specular = float4(1,1,1,1);
    specular *= textureColor;
    specular *= FrontSpecular;
    
    color += LambertFactor(Input.normal, LightDirection0) * LightColor0 * diffuse;
    color += PhongFactor(Input.normal, LightDirection0, FrontShininess) * LightColor0 * specular;
    
    return float4(color.rgb, 1);
    }
  else
    {
    Input.normal = - normalize(Input.normal);
    float4 color = float4(0,0,0,1);
    
    float4 textureColor = tex2D(BackTexture, Input.textureCoordinate);
    
    float4 diffuse = float4(1,1,1,1);
    diffuse *= textureColor;
    diffuse *= BackDiffuse;
    
    float4 specular = float4(1,1,1,1);
    specular *= textureColor;
    specular *= BackSpecular;
    
    color += LambertFactor(Input.normal, LightDirection0) * LightColor0 * diffuse;
    color += PhongFactor(Input.normal, LightDirection0, BackShininess) * LightColor0 * specular;
    
    return float4(color.rgb, 1);
    }
  }

void main(in PS_INPUT Input, in float isBackFacing : VFACE, out PS_OUTPUT Output)
  {
  HandleClipping(Input);
  Output.color = ShadeFragment(Input, isBackFacing);
  }

Just one thing to try :

Try to move the tex2d instructions out of the if()else block. Also since VFACE is a scalar (-1 or 1) it can be used to change the direction of the normal quite easily (ie. multiplying the normal).

(sorry I couldn't resist writing the code in more compact form)

float4 ShadeFragment(PS_INPUT Input, float isBackFacing)
{

float4 FrontColor = tex2D(FrontTexture, Input.textureCoordinate);
float4 BackColor  = tex2D(BackTexture, Input.textureCoordinate);

Input.normal = normalize(Input.normal) * isBackFacing;

float4 color = float4(0,0,0,1);
float4 diffuse;
float4 specular;

if (isBackFacing < 0)
{
    diffuse = FrontColor * FrontDiffuse;
    specular = FrontColor * FrontSpecular;
}
else
{
    diffuse = BackColor * BackDiffuse;    
    specular = BackColor * BackSpecular;
}

color += LambertFactor(Input.normal, LightDirection0) * LightColor0 * diffuse;
color += PhongFactor(Input.normal, LightDirection0, BackShininess) * LightColor0 * specular;
   
return float4(color.rgb, 1);
}

Alternatively change

in float isBackFacing : VFACE
to
in bool isBackFacing : VFACE

float isBackFacing
to
bool isBackFacing

if (isBackFacing < 0)
to
if (!isBackFacing)

and it will work (cmp r0.x, vFace, c8.x, c8.x turns into cmp r0.x, vFace, c8.x, c8.y). Why? I have no idea.

Under SM3, VFACE is a floating point value. In the later (D3D10->) versions you are supposed to use SV_IsFrontFace which a boolean value.

Cheers!

Under SM3, a bool works perfectly too (and apparently better than a float). fxc magic ;)

Thanks for all the suggestions!

When I use the newer compiler, and use bool instead of float, the selection between the two branches seems to work out. But then the normal is still wrong: it gets normalized in both branches, but not flipped for the backside. When I then try the suggestion to calculate the normal by multiplying by VFACE, I get this:

//
// Generated by Microsoft (R) HLSL Shader Compiler 9.30.9200.16384
//
// Parameters:
//
//   float4 BackDiffuse;
//   float BackShininess;
//   float4 BackSpecular;
//   sampler2D BackTexture;
//   float4 FrontDiffuse;
//   float FrontShininess;
//   float4 FrontSpecular;
//   sampler2D FrontTexture;
//   float4 LightColor0;
//   float3 LightDirection0;
//
//
// Registers:
//
//   Name            Reg   Size
//   --------------- ----- ----
//   LightDirection0 c0       1
//   LightColor0     c1       1
//   FrontDiffuse    c2       1
//   FrontSpecular   c3       1
//   FrontShininess  c4       1
//   BackDiffuse     c5       1
//   BackSpecular    c6       1
//   BackShininess   c7       1
//   FrontTexture    s2       1
//   BackTexture     s3       1
//

    ps_3_0
          def c8, 1, -1, 0, 0
          dcl_normal v0.xyz
          dcl_texcoord v1.xy
          dcl vFace
          dcl_2d s2
          dcl_2d s3
   0:     cmp r0.x, vFace, c8.x, c8.y
   1:     mul r0.yzw, -r0.x, v0.xxyz
   2:     nrm r1.xyz, r0.yzww
   5:     if_ne r0.x, -r0.x
   8:       mov r0.xyz, c0
   9:       add r0.xyz, r0, c8.zzxw
  10:       nrm r2.xyz, r0
  13:       dp3_sat r0.x, r1, r2
  14:       pow r1.w, r0.x, c7.x
  17:       texld r0, v1, s3
  17:       mul r2.xyz, r0, c6
  18:       mul r3.xyz, r1.w, c1
  19:       mul r2.xyz, r2, r3
  20:       dp3_sat r0.w, r1, c0
  21:       mul r3.xyz, r0.w, c1
  22:       mul r0.xyz, r0, c5
  23:       mad r0.xyz, r3, r0, r2
  24:     else
  25:       mov r2.xyz, c0
  26:       add r2.xyz, r2, c8.zzxw
  27:       nrm r3.xyz, r2
  30:       dp3_sat r0.w, r1, r3
  31:       pow r1.w, r0.w, c4.x
  34:       texld r2, v1, s2
  34:       mul r3.xyz, r2, c3
  35:       mul r4.xyz, r1.w, c1
  36:       mul r3.xyz, r3, r4
  37:       dp3_sat r0.w, r1, c0
  38:       mul r1.xyz, r0.w, c1
  39:       mul r2.xyz, r2, c2
  40:       mad r0.xyz, r1, r2, r3
  41:     endif
  42:     mov oC0.xyz, r0
  43:     mov oC0.w, c8.x

// approximately 46 instruction slots used (2 texture, 44 arithmetic)

Look at the if statement in line 5! That can't be right?

How come this stuff is so shaky? It's not like I'm doing super advanced stuff... I guess something else must still be wrong. Some more googling points to some forum threads claiming branching code (especially when combined with textures) are not SM3's strong point. I can imagine performance being crappy, but it should still work, right?

Besides, when I use the normal's z-component(in camera space) to decide on which branch to take, everything works fine, even with texturing on both sides.

My guess is the combination of dynamic branching + vface breaks something in the compiler (bug). Try kauna's refactored code (seemed to produce saner asm when i tried it) or try putting a [flatten] in front of the if statement.