I've recently found ugly problems with the d3d11 renderer in my game engine. I'm using IDXGISwapChain with real fullscreen and I have serious problems getting Alt-Tabbing to work. Switching between windowed and full screen mode works completely fine.

That was easy to do with DX8/DX9, where didn't have to do anything but check for a lost device. Alt-tabbing from fullscreen restored the display mode and everything worked fine. Now with the IDXGISwapChain I can't alt-tab out of a game running in fullscreen properly anymore. 

Upon Alt-tabbing the display goes bonkers (e.g. a part of the screen is displayed half offscreen, but also another part in the top left corner (looks like the resolution has changed, and some old graphics are still there in GPU memory). I do not see the desktop at all. Upon clicking anywhere the game is switched to windowed mode, but with the window stripped of all borders (and therefore the client size being not the size I'm expecting). I do not modify the window styles myself (as I did in DX8/DX9), so I suspect IDXGISwapChain. Unfortunately the extra debug output of D3D11 doesn't show any errors.

I've called MakeWindowAssociation with the flag to not affecting anything (not listenen to Alt-Enter or switch modes).

Does anybody have any ideas what I might be doing wrong? Or maybe have a working sample that does windowed and fullscreen toggling with Alt-tabbing working?

Would reacting on WM_ACTIVATEAPP to switch to windowed mode myself be a good solution?

I'm running Windows 10, latest official build. I've tried not to touch WndProc in the DX11 renderer, but I'll probably do that as a last resort.

I remember some similar troubles in my toy engine, which does raw Win32 API calls without relying on SDL or such. On window activation (WM_ACTIVATE message), I manually restore the window if it was minimized before, by calling ShowWindow() with SW_RESTORE argument. Also, in case it's supposed to be fullscreen, I change to fullscreen mode again with ChangeDisplaySettings(). In practice, I don't rely on Direct3D11 to do any magic on my behalf, so the window code would be exactly the same on OpenGL, though I've made the window association to the swapchain. You can check the relevant code here:

https://github.com/cadaver/turso3d/blob/master/Turso3D/Window/Win32/Win32Window.cpp

https://github.com/cadaver/turso3d/blob/master/Turso3D/Graphics/D3D11/D3D11Graphics.cpp

I had some serious issues with switching between windowed and fullscreen mode as well. So I created, I demo project just to get this switching working. Take a look at:

https://github.com/matt77hias/Direct3D-11-Projects

Project 8

Pay attention to:

• https://github.com/matt77hias/Direct3D-11-Projects/blob/master/Direct3D 11/Project 8/Project 8/src/engine.cpp#L221 (before entering the game/rendering loop)
• https://github.com/matt77hias/Direct3D-11-Projects/blob/master/Direct3D 11/Project 8/Project 8/src/engine.cpp#L241 (inside the game/rendering loop)
• https://github.com/matt77hias/Direct3D-11-Projects/blob/master/Direct3D 11/Project 8/Project 8/src/rendering.cpp#L218 (swap chain creation)
• https://github.com/matt77hias/Direct3D-11-Projects/blob/master/Direct3D 11/Project 8/Project 8/src/rendering.cpp#L295 (switching between windowed/fullscreen mode)

The solution is basically the following:

1. Create the swap chain always in windowed mode.
2. If you want to start in fullscreen mode, switch to fullscreen mode after calling ShowWindow.
3. In every iteration of your game/rendering loop, check if you lost the windowed/fullscreen mode (i.e. your tracked windowed/fullscreen mode is not the same as the windowed/fullscreen mode of the swap chain) and check if the user wants to switch (e.g. after pressing ALT+ENTER). If you lost track, synchronize again. If the user wants to switch, switch modes.

Thanks for the help!

I found my issues:

a) There's a flag in ResizeBuffers, which overrides the previously set flags used in CreateSwapChain. I added DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH there as well (as I did in CreateSwapChain). In conclusion: Make sure both flag values match!

b) I'm listening to WM_ACTIVATEAPP now, and when I'm losing focus while in fullscreen I'll switch to windowed mode.