Hi,

I'm seeing a crash inside CreateDevice in an old code base when initializing a d3d device with the D3DCREATE_HARDWARE_VERTEXPROCESSING flag. The d3d initialization is contained in its own thread which is contained in a native dll being called from managed code. I am able to run with D3DCREATE_SOFTWARE_VERTEXPROCESSING or D3DCREATE_MIXED_VERTEXPROCESSING but D3DCREATE_HARDWARE_VERTEXPROCESSING always crashes inside CreateDevice. These flags are always OR'd with D3DCREATE_MULTITHREADED.

There are no error messages returned as the code crashes inside the function. Sometimes in gdi32.dll (and a few times in nvd3dum.dll). It is always a stack overflow exception caught by a chckstck in the thread running the dll. I don't have source for these so don't know whats going on. I'm on the latest NVIDIA drivers and have tried both debug and release d3d9 runtimes (Note: This used to work on older/last year's drivers).

I know the card (GTX 460 SE) supports hardware vertex processing and I am able to get the code with the same creation parameters to work inside the d3d sample demo app. I tried increasing the thread stack size from the default 1mb on windows but this had no effect on the crash either.

Anyone seen anything like this before or know what could be causing this ? or anything else I can do to track the problem ?

Thanks!

Did you anyway (even if you know the card should support HW vertex processing) try to verify it by checking for D3DDEVCAPS_HWTRANSFORMANDLIGHT in GetDeviceCaps? Just for sure ;)

Thanks for the suggestion. Yup, just tried it:

D3DCAPS9 Caps;
sys->GetDeviceCaps(D3DADAPTER_DEFAULT, D3DDEVTYPE_HAL, &Caps);
if( Caps.DevCaps & D3DDEVCAPS_HWTRANSFORMANDLIGHT )
     return true
return false;

returns true.

What is D3DCREATE_HARDWARE_VERTEXPROCESSING anyway ? Does this mean the entire geometry stage (vertex fetch, vertex shader etc) runs in software or just the old fixed function stages ? I can even get it to work with D3DCREATE_MIXED_VERTEXPROCESSING which makes the CPU run a lot faster so I'm assuming its doing all the heavy lifting on the gpu. I just don't know why its crashing with D3DCREATE_HARDWARE_VERTEXPROCESSING.

Hm that's a good question :) I also don't know what exactly does HARDWARE_VERTEXPROCESSING include. Vertex transformations and fixed-function lighting for sure, that's the good old HW T&L, I remember when this came out and some games then started to require GPUs capable of it :)

Probably if you use D3DCREATE_SOFTWARE_VERTEXPROCESSING with a vertex shader, then the shader is executed on the CPU?

Yeah, that's what I'm assuming. Software VP would mean vertex shader emulated on the CPU whereas mixed picks hardware if it can otherwise fall backs to software ?

Maybe that's why it fixed my crash, by delaying some initialization to happen sometime later which causes the crash to not happen.

If you are getting a stack overflow, increase the size of the stack.

Thanks for the reply mhagain. I did take a quick look and the code around the device creation looked ok, but I'll check again ( I also tried increasing the thread stack just as a test but I still got the crash, so I was thinking something might just be blowing the stack somewhere (infinite recursion, or huge allocation on the stack) ). Yeah, I'm assuming the fault is in our code and not in the drivers. I got the sample app to work correctly, however I haven't tried spawning a thread from a native dll called from managed code and creating a device from there yet.

I was curious if the CreateDevice function might have some callbacks into user code ? If so, what would these callbacks be. for eg: the d3dxeffect system has user callbacks which get called from inside the d3d dll's. Maybe create device is calling such a callback which is causing the stack overflow. If so what kind of usage would trigger a callback from inside CreateDevice ? I can't step into this function so I'm not sure how to find out.

actually its probably not a callback blowing the stack as I would see it in the callstack, still could be a stomp though.