Recently I bumped into this weird problem when I tried to compile a HLSL pixel shader. The D3DCompile function just did a infinite recursive call and caused a stack overflow. I simplified the pixel shader as following:
uniform float end : register(c0);
uniform float start : register(c1);
static float4 c[1] =
{
float4(0, 0, 0, 0)
};
struct PS_INPUT
{
float4 v0 : TEXCOORD0;
};
struct PS_OUTPUT
{
float4 gl_Color0 : SV_TARGET0;
};
PS_OUTPUT main(PS_INPUT input)
{
c[0] = input.v0;
if (c[0].y > start && c[0].y < end)
{
c[0] *= 1.0;
}
else if (c[0].y >end)
{
c[0] *= 2.0;
}
PS_OUTPUT output;
output.gl_Color0 = c[0];
return output;
}
Then, I tried to compile it with D3DCompile function as following:
HMODULE mD3DCompilerModule = LoadLibrary(D3DCOMPILER_DLL);
auto mD3DCompileFunc = reinterpret_cast<pD3DCompile>(GetProcAddress(mD3DCompilerModule, "D3DCompile"));
mD3DCompileFunc(s, strlen(s), "D:\\fakepath", NULL, NULL, "main", "ps_5_0", 0, 0, NULL, NULL);
which is pretty standard.
The crash(not a failure return value) happend at the last line, in d3dcompiler_47.dll.
Though I am new to DirectX, I think my shader should be correct with grammar, so what is wrong with this example?
Some cases that will make the compilation successful that I found:
- using a static varaible c instead of c[1] array.
- using 'else' instead of 'else if' in the condition control flow, however the semantics is changed.
- using D3DCOMPILE_SKIP_OPTIMIZATION when compiling, losing binary optimization
