Drivers do this to enable very quick paging of GPU memory. If the OS tells your application that it needs all the GPU memory for another process (e.g. the user just alt+tabbed to chrome), the driver can instantly discard all your textures in the GPU, safe in the knowledge that it still has a copy of them in system RAM that it can use to recreate them when you alt+tab back into the game. You're kind of at the mercy of your GPU drivers here.

You could try using D3D11_BIND_RENDER_TARGET, but even if it does trick the driver into not holding onto a system memory cache/copy, it will have potential performance impacts of its own...

I would echo what galo1n mentioned about being careful with regards to which memory statistic you're looking at. Like any OS that uses virtual memory. Windows has quite a few statistics that you can query via task manager or Win32 API's and they all have different meanings (and sometimes the difference is quite subtle).

D3D resources will always allocate committed virtual memory via VirtualAlloc. This will increase the "commit size" in task manager, which is telling you the total amount of committed virtual memory in the process. By extension it will also increase the system total commit size that you can view under the "Performance" tab of task manager (it's the value labeled "Committed"). If you want to query these values programatically, you can use GetProcessMemoryInfo for process-specific stats and GetPerformanceInfo for system-wide stats.

Committed memory has to be backed up by either system memory or the page file, which means that your commit total is typically not equal to the physical RAM consumption of your process. To see that, you want to look at the private working set, which is visible in task manager under the Details and Performance tabs. The process and system totals are also returned by the functions I linked above. In general your working set will be a function of how much memory your program is actually accessing at any given time. So if you allocate a bunch of memory that you never use, it can get paged out to disk and it won't be reflected in your working set. However Windows tends to only page out data when it really needs to, so if you access some memory once and then never again, it can stay in your working set until somebody needs that physical memory.

If your D3D resources are causing a large increase in your working set, then it's possible that you're over-committing the GPU's dedicated memory pool. Either the Windows video memory manager (VidMM) or the video card driver (depending on the version of Windows that you're running) will automatically move GPU data to and from system memory if it can't keep the entire system's worth of resources resident in dedicated GPU memory. If this happens a lot it can really tank your performance, so you generally want to avoid it as much as possible. You can check for this by capturing your program with ETW and using GPUView, or by using the new PIX for Windows.

In general though you probably want to limit your committed memory as much as possible, even if your working set is low. Like I mentioned earlier it needs to be backed up by either system memory or the page file, and if those are both exhausted your system will start to get very unhappy and either you or the video card's user-mode driver will crash and burn. This unfortunately means that the system's physical memory amount, page file size, and memory usage of other programs will dictate how much memory you can use without crashing, which in turn means that your higher performance settings may crash even if they have a nice GPU with lots of dedicated memory. On the last game I shipped there were a non-trivial number of crashes from users with high-end video cards who cranked their settings, but had their page file turned off! 

You should already be using Texture Compression, right? If you aren't, then GPU performance is going to be suffering.

BC1 will give you the smallest footprint for a given resolution, half that of BC3 or BC7 for a moderate loss of quality.

1 minute ago, Holy Fuzz said:

it's a 2D pixel art game. Every texture compression format I've tried makes it look pretty terrible. It's also not anywhere close to being GPU-bound.

Well, if you aren't using any at all yet, then BC7 is probably the best place to start. It's 4x smaller then an uncompressed RGBA8 texture and the quality from a good compressor like ISPC is pretty good.

BC1 is 8x smaller than an uncompressed texture, so you could even afford to double the dimensions on every texture and still end up with a texture that's 2x smaller than the original uncompressed-but-lower-resolution version.