So, first things first:

Your application should never change the display mode unless I ask it to. This is 2012, you don't need to start your game at 640x480. Launch it at the resolution my screen is currently running at, as that's my preferred resolution. The number of games that, even today, start the game by default in a reduced resolution is significant. It irritates me greatly, especially since I'm on a multi-monitor system and it FUCKS UP the windows on the secondary monitor.

You should assume decent defaults, optionally you could run a performance test on first startup that makes a decent guess at the appropriate settings. Make sure to inform the customer of this though, don't just start changing resolutions and running demos to test performance.

Allow the user to change the settings. Either in game or a launcher control panel.

Support Full-screen Windowed mode. This is a window (typically WS_POPUP) that is NOT WS_EX_TOPMOST and does not involve a resolution change. It's a windowed application that takes up the entire desktop. GetDesktopWindow, GetWindowRect can be used to appropriately size the window.

You should always start up on the primary display unless I specify otherwise. If your application is in windowed mode and they drag it around then there are messages you can trap to change it to the appropriate device, although the windowed device is typically virtualized somewhat so its not that big of an issue.

All modern APIs (read DX) support mechanisms for querying the display modes, bit depths, texture modes, and anti-aliasing techniques supported by the card. You can use this to tailor your presented options correctly.

Yes. A game that is fullscreen-windowed can be trivially alt tabbed between say a "guide" and the game. Or if you have other windows open on other screens your are not constantly changing resolution/flickering every time you alt+tab in and out of the game.

From the player's perspective it makes a lot more sense. They not longer go "what the fuck? I hit some key on my keyboard and the entire game vanished, where did it go?"

Ok, so if I am going to start up in fullscreen windowed mode, do I still give the user the option to go to windowed mode? If so, what would my app do, just change the window style so that the title bar and borders are displayed?

Yes, you should ideally still allow for traditional windowed mode, as some people prefer it that way as well.

Currently I am starting off with a WS_POPUP window. When the user hits 'W', I run the following code:


SetWindowLongPtr(g_hWnd, GWL_STYLE, WS_OVERLAPPED);
SetWindowPos(g_hWnd, HWND_TOP, curWindowPosX, curWindowPosY, windowedWidth, windowedHeight, SW_SHOW);

(curWindowPosX and curWindowPosY are both set to 50, and never changed. windowedWidth and windowedHeight are set to 640 and 480 respectively).

But it seems to have no effect. And if I click the window after running that code, it disappears. Any advice on how to switch to a true windowed mode from a fullscreen windowed mode?
[/quote]
Try something more like:
#include <windows.h>
#include <atlbase.h>
#include <atlwin.h>

class MyWindow : public CWindowImpl<MyWindow, CWindow, CWinTraits<WS_POPUP, 0>>
{
public:
DECLARE_WND_CLASS_EX(NULL, CS_OWNDC | CS_HREDRAW | CS_VREDRAW, reinterpret_cast<HBRUSH>(BLACK_BRUSH));

MyWindow() {
auto desktop = ::GetDesktopWindow();
RECT bounds;
::GetWindowRect(desktop, &bounds);

Create(0, bounds, _T("COOKIES!"));
}

private:
BEGIN_MSG_MAP(MyWindow)
MESSAGE_HANDLER(WM_DESTROY, OnDestroy);
MESSAGE_HANDLER(WM_KEYDOWN, OnKeyDown);
END_MSG_MAP()


LRESULT OnKeyDown(UINT msg, WPARAM wParam, LPARAM lParam, BOOL& result) {
if(wParam == VK_RETURN) {
RECT bounds = {50, 50, 800, 600};
LONG_PTR p = GetWindowLongPtr(GWL_STYLE);
if(!(p & WS_POPUP)) {
auto desktop = ::GetDesktopWindow();
::GetWindowRect(desktop, &bounds);
}
p ^= WS_OVERLAPPEDWINDOW;
p ^= WS_POPUP;
SetWindowLongPtr(GWL_STYLE, p);
SetWindowPos(0, bounds.left, bounds.top, bounds.right, bounds.bottom, SWP_NOZORDER | SWP_NOOWNERZORDER);
} else {
result = FALSE;
}
return 0;
}

LRESULT OnDestroy(UINT msg, WPARAM wParam, LPARAM lParam, BOOL& result) {
PostQuitMessage(0);
return 0;
}
};

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR cmdLine, int nCmdShow) {
MyWindow w;
w.ShowWindow(nCmdShow);
w.UpdateWindow();

MSG msg;
while(GetMessage(&msg, 0, 0, 0)) {
TranslateMessage(&msg);
DispatchMessage(&msg);
}
return 0;
}

Which works fine for me.

Thanks for the code, Washu.

It took me hours to realize that I was using "WS_OVERLAPPED" where I should've used WS_OVERLAPPEDWINDOW.

I will still take a look at your code, though. Looks like it has a nice, clean architecture.

It's more than that. You should not simply pass in WS_OVERLAPPED or WS_OVERLAPPEDWINDOW, as other window styles are often appended internally. This is why my code uses xor-assignment to add/remove the appropriate flags from the ACTUAL window style code.