Sign in to follow this  
Curno

D3DCompile crashes with stack overflowing when compiling this HLSL. Very Strange

Recommended Posts

Recently I bumped into this weird problem when I tried to compile a HLSL pixel shader. The D3DCompile function just did a infinite recursive call and caused a stack overflow. I simplified the pixel shader as following:

uniform float end : register(c0); 
uniform float start : register(c1); 
static float4 c[1] =
{
    float4(0, 0, 0, 0)
};
struct PS_INPUT
{
    float4 v0 : TEXCOORD0;
};
struct PS_OUTPUT
{
    float4 gl_Color0 : SV_TARGET0;
};  
PS_OUTPUT main(PS_INPUT input)
{
    c[0] = input.v0; 
    if (c[0].y > start && c[0].y < end)
    {
        c[0] *= 1.0; 
    }
    else if (c[0].y >end)
    {
        c[0] *= 2.0; 
    }
    PS_OUTPUT output; 
    output.gl_Color0 = c[0]; 
    return output;
}

Then, I tried to compile it with D3DCompile function as following:

HMODULE mD3DCompilerModule = LoadLibrary(D3DCOMPILER_DLL);
auto mD3DCompileFunc = reinterpret_cast<pD3DCompile>(GetProcAddress(mD3DCompilerModule, "D3DCompile"));
mD3DCompileFunc(s, strlen(s), "D:\\fakepath", NULL, NULL, "main", "ps_5_0",  0, 0, NULL, NULL);

which is pretty standard.

The crash(not a failure return value) happend at the last line, in d3dcompiler_47.dll.

Though I am new to DirectX, I think my shader should be correct with grammar, so what is wrong with this example?

Some cases that will make the compilation successful that I found:

  1. using a static varaible c instead of c[1] array.
  2. using 'else' instead of 'else if' in the condition control flow, however the semantics is changed.
  3. using D3DCOMPILE_SKIP_OPTIMIZATION when compiling, losing binary optimization

Share this post


Link to post
Share on other sites

Unfortunately shader compiler crashes are something you have to deal with as a graphics programmer. If you want to report the bug, you can try doing that at the official DirectX forums. However they're mostly focused on the new compiler these days, so I wouldn't count on it getting fixed anytime soon. If you have a workaround that gets you the results you want, I would suggest that you use it.

 

By the way, the declaration of "start" and "end" are using the old-style uniform declaration from DX9. Declaring global uniform variables is still legal with the newer shader models, but ultimately everything has to get packed into the constant buffers. If you just declare global uniforms like you did, they're get placed into an implicit constant buffer called "$Globals". If you'd explicity define your own constant buffer you can do it like this:

 

cbuffer Constants : register(b0)
{
    float end;
    float start;
}

 

Note that if you continue declaring them as global uniforms, the register assignment will have no effect since uniforms are no longer bound to individual constant buffers.

Share this post


Link to post
Share on other sites

Adding to what MJP says, I find it quite odd that you have c[1] declared as a static global; it certainly doesn't need to be (it's only ever referenced inside your main shader function) and I'm wondering if your intention is that you actually want it's value to persist across multiple shader invocations (in which case a static global is certainly not what you actually want).  I suggest that you should make it a local and see what your compiler behaviour is then.

Share this post


Link to post
Share on other sites

Thanks for all the reply. As I mentioned in the post, I just simplified the original shader I used to make my point. I just want to make sure this is a compiler bug instead of problemof my shader. 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this