Before creating my main window and swap chain (in Windows 8.1 and Direct3D 11), my application displays some graphics settings the user can configure such as choosing between windowed and full screen mode.

Based on the outcome, I create my main window via CreateWindow with the (WS_OVERLAPPED | WS_CAPTION | WS_SYSMENU | WS_MINIMIZEBOX) or WS_POPUP flags.

My DXGI_SWAP_CHAIN_DESC1 looks like:

swap_chain_desc.Width = width;
swap_chain_desc.Height = height;
swap_chain_desc.Format = g_device_enumeration->GetDisplayMode()->Format;
swap_chain_desc.SampleDesc.Count = 1;
swap_chain_desc.SampleDesc.Quality = 0;
swap_chain_desc.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT;
swap_chain_desc.BufferCount = 1;

dxgi_factory3->MakeWindowAssociation(m_hwindow, DXGI_MWA_NO_ALT_ENTER);

swap_chain_desc.Flags = DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH; 

You can always update the window style at the transition, but usually, you can just ignore that as in fullscreen, windows enforce a no border style.

Does D3D or DXGI provide a way to specify the behavior in case of an ALT+ENTER press or do I check this myself?

Ok, so I added this again:

swap_chain_desc.Flags = DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH;

and added this which is new for me (thank you):

dxgi_factory3->MakeWindowAssociation(m_hwindow, DXGI_MWA_NO_WINDOW_CHANGES);

WS_OVERLAPPED | WS_CAPTION | WS_SYSMENU | WS_MINIMIZEBOX

You do not want to let Windows deal with alt enter and windows change. Any serious windowed/fullscreen switch code does it itself to control the process. So first :

MakeWindowAssociation(hwnd, DXGI_MWA_NO_WINDOW_CHANGES | DXGI_MWA_NO_ALT_ENTER | DXGI_MWA_NO_PRINT_SCREEN);

Then in your message loop :

case WM_MENUCHAR:
        return MNC_CLOSE<<16; // prevent windows to beep on alt enter from fullscreen
case WM_SYSKEYDOWN:
	if (wParam==VK_RETURN) {
		window->SetToggleScreen(); // code that set a flag in your app to signal that a request has been done
		return 0;
	}
	return DefWindowProc(hwnd,uMsg,wParam,lParam);

Then, you have to monitor for alt+tab and similar because you cannot prevent them to force you to leave fullscreen, so at the beginning of your app render loop.

// handle fullscreen switch
bool altTab = sc_.LostFS();
if (wnd_.ToggleScreen()||altTab) {
	sc_.Toggle(device_,!altTab);
	wnd_.ClearToggleScreen();
}

bool SwapChain::LostFS() {
	BOOL current = false;
	sc_->GetFullscreenState(&current, nullptr);
	return current == false && isFS_ == true;
}
void SwapChain::Toggle(Device& device, bool toggle, Output* output ) {
	buffers_.clear(); // Clear any reference to swap chain buffers or ResizeBuffers will fail

	BOOL current = false;
	if (toggle) { // the toggle case, skipped in case of alt tab as we just want to recreate the buffers
		sc_->GetFullscreenState(&current, nullptr);
		current = !current;
		sc_->SetFullscreenState(current, output ? output->Get() : nullptr );
	}
	sc_->ResizeBuffers(desc_.BufferCount, desc_.Width, desc_.Height, desc_.Format, DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH);
	CreateRTVs(device); // recreate references

	sc_->GetFullscreenState(&current, nullptr); // windows may deny the change, test the current status
	isFS_ = current != 0;
}

Thanks for the lost full screen situation. I was already wondering, while debugging, what I was doing wrong since I couldn't keep track of the fullscreen/window status myself. Once my renderloop started, my manual status seemed to be diverged for no clear reason.

At first, I also had some issues with calls to SetFullscreenState failing due to an 0x887a0004 error. I also got this error for creating a device if a non-nullptr pAdapter was passed. A bit strange though, since when I pass a nullptr and retrieve the adapter afterwards, the descriptor said it was the same adapter anyway. So apparently I now also pass D3D_FEATURE_LEVEL_11_0 instead of only D3D_FEATURE_LEVEL_11_1 for my GTX 960M (which was not necessary for my GTX 970).

So it now works nearly fine. Somehow I need to press enter twice (quickly enough), but I am still looking into it.

Fix the problem (although still figuring out whether the solution is "correct")

Instead of your:

return current == false && isFS_ == true;

or (in my code):

return  m_fullscreen && IsWindowed();

I check for a full screen or windowed loss (i.e. m_fullscreen does not match the full screen state of the swap chain):

return  m_fullscreen == IsWindowed();

Message delegate:

case WM_SYSKEYDOWN: {
    if (wParam == VK_RETURN) {
        g_engine->SetModeSwitchFlag(true);
        return 0;
    }
    return DefWindowProc(hWnd, msg, wParam, lParam);
}

Renderer setup:

dxgi_factory3>MakeWindowAssociation(m_hwindow, DXGI_MWA_NO_WINDOW_CHANGES | DXGI_MWA_NO_ALT_ENTER | DXGI_MWA_NO_PRINT_SCREEN);
...
swap_chain_desc.Flags              = DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH;

Renderer setup (after swap chain creation but before RTV creation):

// Set mode (full screen or windowed).
BOOL current = g_device_enumeration->IsFullScreen();
m_swap_chain2->SetFullscreenState(current, nullptr);
m_swap_chain2->GetFullscreenState(&current, nullptr);
m_fullscreen = (current != 0);

Render loop:

// Handle switch between full screen and windowed mode.
const bool lost_full_screen = m_renderer->LostFullScreen();
if (m_mode_switch || lost_full_screen) {
    m_renderer->SwitchMode(!lost_full_screen);
    m_mode_switch = false;
}

Switch mode:

void Renderer::SwitchMode(bool toggle) {

        // Release the swap chain buffers.
        m_render_target_view->Release();

        BOOL current = false;
        if (toggle) {
            m_swap_chain2->GetFullscreenState(&current, nullptr);
            current = !current;
            m_swap_chain2->SetFullscreenState(current, nullptr);
        }

        // Recreate the swap chain buffers.
        m_swap_chain2->ResizeBuffers(0, 0, 0, DXGI_FORMAT_UNKNOWN, DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH);
        SetupRenderTargetView();

        m_swap_chain2->GetFullscreenState(&current, nullptr);
        m_fullscreen = (current != 0);
    }

Windowed mode > Full screen > Windowed mode > Full Screen > ... works fine.

Full screen > Windowed mode: results in a white client area on the window which I cannot control anymore > Full screen: works fine

You should test every function error code to find where things get bad. One thing tho, Microsoft does not recommend to initialize directly in fullscreen, because as you know, it can be denied, and other unpleasant behavior. The proper way to start fullscreen is to start windowed and then call SetFullscreenState. You can keep your window hidden to not see it pop for an instant.

I read that on MSDN but doesn't this only apply for a DXGI_SWAP_CHAIN_DESC which has an explicit Windowed member variable?

I use a DXGI_SWAP_CHAIN_DESC1 which does not seem to support full screen while creating the swap chain. So therefore, I do:

BOOL current = g_device_enumeration->IsFullScreen();
m_swap_chain2->SetFullscreenState(current, nullptr);
m_swap_chain2->GetFullscreenState(&current, nullptr);
m_fullscreen = (current != 0);

after the swap chain creation.