I'm seeing a crash inside CreateDevice in an old code base when initializing a d3d device with the D3DCREATE_HARDWARE_VERTEXPROCESSING flag. The d3d initialization is contained in its own thread which is contained in a native dll being called from managed code. I am able to run with D3DCREATE_SOFTWARE_VERTEXPROCESSING or D3DCREATE_MIXED_VERTEXPROCESSING but D3DCREATE_HARDWARE_VERTEXPROCESSING always crashes inside CreateDevice. These flags are always OR'd with D3DCREATE_MULTITHREADED.
There are no error messages returned as the code crashes inside the function. Sometimes in gdi32.dll (and a few times in nvd3dum.dll). It is always a stack overflow exception caught by a chckstck in the thread running the dll. I don't have source for these so don't know whats going on. I'm on the latest NVIDIA drivers and have tried both debug and release d3d9 runtimes (Note: This used to work on older/last year's drivers).
I know the card (GTX 460 SE) supports hardware vertex processing and I am able to get the code with the same creation parameters to work inside the d3d sample demo app. I tried increasing the thread stack size from the default 1mb on windows but this had no effect on the crash either.
Anyone seen anything like this before or know what could be causing this ? or anything else I can do to track the problem ?