Hey,

Unfortunately I cannot find my answer in the already existing topics, so here I come. I have some problems with the windowing system, especially with chaning the window style at runtime.

First of all, the window has 3 different mode: Windowed, Borderless and Fullscreen. Borderless and Fullscreen mode means the window's style is simply a WS_POPUP, while the Windowed mode means the window can have borders, system menu, maximize button, and so on. The window has border, caption, minimize-maximize-close buttons by default when selecting Windowed mode.

I'm using the WM_SIZE message to resize the frame buffers (aka render targets). The window does not have a resizing border, so only the engine (and the maximize/restore buttons) can change the size and cause the message to arrive.

I have a function in my Window class which supposed to be used to change the style and/or the resolution of the window. And this is not working. :) Initially, I'm creating a window with the startup settings. Now it's 1280x720 windowed = border + caption. The size is adjusted, so the client area is the 1280x720, the window itself is bigger.

For testing purposes, I'm using the F1-F3 keys to select the new window mode. The resolution is the same for now. I'm also running FRAPS to check if everything is okay. Here are the steps:

0) the window is shown, everything is okay (I have a really simple test scene)

1) switch to borderless mode: it looks like the style is changed (the borders are gone), however the FPS counter is gone and I have a strange "border" at the left and top side of the window. So it seems the style change was not really successful.

2) switch back to windowed mode: everything works again, the FPS counter is back, the window border is back, and the viewport fits.

3) switch to borderless mode again: this is the worst case - a double resize event is raised, first with size 1296x758 (which is the full size of the window with borders) then back to 1280x720. This means a double-resize of textures but the size actually isn't changed. And of course the same problems as in the 1st step.

And here is the code of the window changing function:

void WindowsWindow::Reshape(uint32 width, uint32 height, WindowMode mode)
{
    // change style based on window mode change
    if (windowMode != mode)
    {
        windowMode = mode;

        //const uint32 styleChangeFlags = SWP_NOMOVE | SWP_NOSIZE | SWP_NOZORDER | SWP_FRAMECHANGED;

        if (mode == WindowMode::Fullscreen || mode == WindowMode::Borderless) // going full-screen
        {
            SetWindowLongPtr(hwnd, GWL_STYLE, fullscreenStyle);
            //SetWindowPos(hwnd, nullptr, 0, 0, 0, 0, styleChangeFlags);
        }
        else // going windowed
        {
            SetWindowLongPtr(hwnd, GWL_STYLE, windowedStyle);
            //SetWindowPos(hwnd, nullptr, 0, 0, 0, 0, styleChangeFlags);
        }
    }

    uint32 flags = SWP_NOZORDER | SWP_NOACTIVATE | SWP_FRAMECHANGED | SWP_SHOWWINDOW;

    int32 x = 0;
    int32 y = 0;

    // adjust window size and position
    if (windowMode == WindowMode::Windowed)
        flags |= SWP_NOMOVE;
    Adjust(x, y, width, height);

    SetWindowPos(hwnd, nullptr, x, y, width, height, flags);
}

I hope someone can help me solve this issue.

Edit:
The code was just changed that's why I have comments and "strange if" there. :)

Edit 2:

If I change the window mode to borderless, I get a window which looks like as in the 2nd image above. However if after this I change the resolution to ... say 1024x768 then back to 1280x720, everything works fine. (Haven't tested fullscreen mode yet, that will be another problem)

I don't know about OpenGL, but under D3D11 I resolved similar symptoms by not actually responding directly to WM_SIZE but instead just setting a flag that a size change was requested. Then at the start of the next frame I check this flag and actually implement the size change there (clearing the flag when done).

I though something similar, thanks for the feedback!

It solves the resizing issue (which is irrelevant from the graphics API) but unfortunately the strange behavior remains :(

Thanks Erik, I'll check your solution soon and if it works, I'll try to find the difference which makes it work. :)

Edit:
An interesting difference: when creating a window I define a style based on the parameters. The return value of SetWindowLongPtr is the previous window style which should be the same as the manually defined style, but it's not. Is that normal?

I've shortened (and copy-pasted) the code and tested it:

void WindowsWindow::Reshape(uint32 width, uint32 height, WindowMode mode)
{
    if (mode != WindowMode::Windowed)
    {
        windowMode = mode;

        int32 x = 0;
        int32 y = 0;
        Adjust(x, y, width, height);

        SetWindowLongPtr(hwnd, GWL_STYLE, WS_CLIPCHILDREN | WS_CLIPSIBLINGS | WS_OVERLAPPED | WS_VISIBLE);
        SetWindowLongPtr(hwnd, GWL_EXSTYLE, 0);
        SetWindowPos(hwnd, HWND_TOPMOST, x, y, width, height, SWP_FRAMECHANGED | SWP_DRAWFRAME);

        return;
    }

    // ...
}

This is called when I try to switch to borderless mode. The adjust function is nothing special:

void WindowsWindow::Adjust(int32& x, int32& y, uint32& width, uint32& height)
{
    if (windowMode == WindowMode::Windowed)
    {
        if ((windowedStyle & WS_BORDER) != 0) // hasBorder
        {
            RECT r = { x, y, x + width, y + height }; // left, top, right, bottom
            AdjustWindowRectEx(&r, windowedStyle, 0, extendedStyle);

            x = r.left;
            y = r.top;
            width = r.right - r.left;
            height = r.bottom - r.top;
        }
    }
    else
    {
        HMONITOR monitor = MonitorFromWindow(hwnd, MONITOR_DEFAULTTONEAREST);
        MONITORINFO monitorInfo;
        monitorInfo.cbSize = sizeof(MONITORINFO);
        ::GetMonitorInfoA(monitor, &monitorInfo);

        const int32 monitorWidth = monitorInfo.rcMonitor.right - monitorInfo.rcMonitor.left;
        const int32 monitorHeight = monitorInfo.rcMonitor.bottom - monitorInfo.rcMonitor.top;

        // use the width/height <= monitor's width/height
        width = Math::Min<int32>(monitorWidth, width);
        height = Math::Min<int32>(monitorHeight, height);

        // use the monitor's left/top
        x = monitorInfo.rcMonitor.left;
        y = monitorInfo.rcMonitor.top;
    }
}

And the problem is the same, strange un-updated window are appears. Maybe it's not a WinAPI problem? Or do I miss a message which should be handled? It seems like the OpenGL does not know that the window is changed. After I actually resize the window back and forth (1280x720 -> 1024x768 -> 1280x720), everything is fine.

The loop is pretty standard, this is called every frame:

void WindowsApplication::PumpMessages(const float32 dt)
{
    MSG msg = { 0 };

    while (PeekMessageA(&msg, nullptr, 0, 0, PM_REMOVE))
    {
        TranslateMessage(&msg);
        DispatchMessageA(&msg);
    }
}

LRESULT WindowsApplication::ProcessMessage(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam)
{
    static ModifierKeys modifiers;

    Windows::iterator it = windows.find(hwnd);
    GenericWindowPtr window = (it != windows.end()) ? it->second : nullptr;

    switch (msg)
    {
        // alt+f4 or X button
    case WM_CLOSE:
        REQUIRE(window != nullptr);
        messageHandler.OnWindowClose(window);
        return 0;

    case WM_DESTROY:
        REQUIRE(window != nullptr);
        windows.erase(hwnd);
        return 0;

    case WM_SIZE:
    {
        const int32 w = LOWORD(lparam);
        const int32 h = HIWORD(lparam);

        switch (wparam)
        {
        case SIZE_MINIMIZED:
            // minimized
            break;

        case SIZE_MAXIMIZED: // fall through
        case SIZE_RESTORED:
            if (window != nullptr)
                messageHandler.OnWindowResized(window, w, h);
            break;

        default:
            break;
        }
    }
        break;

    case WM_ACTIVATE:
    {
        REQUIRE(window != nullptr);
        const bool activated = (LOWORD(wparam) != WA_INACTIVE);
        messageHandler.OnWindowActivationChanged(window, activated);
    }
        return 0;

    // tons of input handler

    default:
        break;
    }

    return DefWindowProc(hwnd, msg, wparam, lparam);
}

MSDN on SetWindowLong states:

Specifically, if you change any of the frame styles, you must call SetWindowPos with the SWP_FRAMECHANGED flag for the cache to be updated properly.