DX9 Doubling Memory Usage in x64

D3D9RenderImpl.cpp

Why do you think it's strictly about d3d? It might be your data.

If you operate with lots of pointers in structures - those will double their size in x86-64.

You'd better first check if it's really d3d data (with 'pix' maybe), or yours.

Adding to what vstrakh suggested, your VERTEX structure could be responsible if it uses some datatype that expands, such as 32-bit floats on x86 to 64-bit floats on x64. If you have big vertex buffers that would potentially double their total size then.

I don't have lots of pointers. What takes a lot of memory are video frame buffers and DX9-related objects so I'm pretty sure the difference is related to that.

As for VERTEX... I honestly don't understand how they work. They just have to be set up that way for the code to work with plain 2D frames. Could this really be causing so much memory usage? If so, can it be worked around?

Adding to what vstrakh suggested, your VERTEX structure could be responsible if it uses some datatype that expands, such as 32-bit floats on x86 to 64-bit floats on x64. If you have big vertex buffers that would potentially double their total size then.

You're saying that FLOAT becomes 64-bit instead of 32-bit? That could definitely be a problem.

Searching Google, however, doesn't show anything that indicates FLOAT to behave any differently on x64 platform. Float is 32-bit, Double is 64-bit, and from what I'm reading, it remains the same.

Although C++ don't clearly define the size of floating-point data types, Microsoft has it well-defined here

I just ran some quick test on the ConvertToShader and ConvertFromShader code alone (running 3 times), without running through DX9 shaders.

x86 gave a memory usage of 28MB and 153fps. x64 gave a memory usage of 27MB and 185fps. No problem here. Which means the problem is definitely related to the code executing DX9 shaders.

I can guess at one possibility. Some time ago Microsoft optimized the address space usage of D3D9 on Vista - see https://support.microsoft.com/en-gb/kb/940105 It's possible that that optimization was only applied to x86 as you're not going to run out of address space on x64.

Is this extra memory usage actually causing a significant, measurable performance issue? If not I wouldn't worry about it.

If you really want to investigate what's going on, I'd suggest creating the simplest possible test program that shows the memory usage difference, and using a tool like https://technet.microsoft.com/en-us/sysinternals/vmmap.aspx to investigate how memory gets allocated differently.

I have run VMMap. Here is the result.

32-bit

Type         Size        Committed   Private   Total WS  Private WS  Shareable WS  Shared WS  Locked WS  Blocks  Largest     
Total        1,332,764   1,092,604   910,776   549,472   511,088     38,384        11,308                2701    
Image        196,132     195,828     24,736    44,500    8,752       35,748        8,728                 616     39,016
Mapped File  4,956       4,956                 456                   456           448                   4       3,292
Shareable    25,840      5,780                 2,176                 2,176         2,128                 39      20,480
Heap         686,452     660,384     660,384   331,516   331,516                                         1042    16,192
Managed Heap                                                                                                     
Stack        167,680     9,084       9,084     4,008     4,008                                           786     1,024
Private Data 210,428     192,480     192,480   142,724   142,720     4             4                     214     8,192
Page Table   24,092      24,092      24,092    24,092    24,092                                                  
Unusable     17,184                                                                                              60
Free         2,885,568                                                                                   66      2,079,936

64-bit

Type         Size              Committed   Private     Total WS    Private WS  Shareable WS  Shared WS  Locked WS  Blocks  Largest           
Total        2,719,536         2,496,020   2,274,976   1,922,640   1,874,980   47,660        11,260                2405    
Image        232,656           232,656     22,280      48,456      3,560       44,896        8,588                 723     46,888
Mapped File  4,956             4,956                   476                     476           432                   4       3,292
Shareable    25,708            5,648                   2,280                   2,280         2,232                 35      20,480
Heap         735,564           705,844     705,780     385,196     385,192     4             4                     761     16,192
Managed Heap                                                                                                               
Stack        139,264           5,520       5,520       3,888       3,888                                           408     1,024
Private Data 1,552,624         1,535,908   1,535,908   1,476,856   1,476,852   4             4                     474     12,384
Page Table   5,488             5,488       5,488       5,488       5,488                                                   
Unusable     23,276                                                                                                        60
Free         137,436,239,360                                                                                       64      137,393,457,984

I have also tried running it with no extra memory available, and it didn't reduce the memory usage of this process.

Any idea from here?

Adding to what vstrakh suggested, your VERTEX structure could be responsible if it uses some datatype that expands, such as 32-bit floats on x86 to 64-bit floats on x64. If you have big vertex buffers that would potentially double their total size then.

The VERTEX structure contains floats, which are 32bit on both platforms.

Looking at the VMMap data, the main difference between the two is in the "Private Data" section - there's over 1GB extra in there in the x64 version.

According to the VMMAP documentation:

Private memory is memory allocated by VirtualAlloc and not suballocated either by the Heap Manager or the .NET run time. It cannot be shared with other processes, is charged against the system commit limit, and typically contains application data.

Assuming you're not calling VirtualAlloc() directly yourself, it's probably allocated by either D3D or the graphics driver.

In addition, if you look at the details of that private data, it's made up of large numbers of small allocations. On x64 there are twice as many of them, and they are averaging about three times as big (210,428 / 214 = 983.3; 1,552,624 / 474 = 3275.5).

It could either be memory leaks (which the debug runtime should complain about), or there could be a difference in behaviour between the two.

I believe WPA / XPERF should be able to give you call stacks for the VirtualAlloc calls, but I'm not sure on the details. You could also try breakpointing it in the debugger.

The code itself is fairly simple, behaves the same way and provides the same output.

I'm 99% sure that the difference is in the way DX9 or the video driver handles its memory internally.

Yet knowing this doesn't solve the problem.

What eats up memory are many texture buffers to process video frames through various steps of processing. The memory gets allocated during initialization by calling CreateInputTexture and gets released only after it is done processing all the frames.

I tried creating the device with D3DCREATE_DISABLE_PSGP_THREADING but that doesn't help.

I run it through Visual Studio 2015 debugger to analyze the memory allocations. Here is the result.

Although the process is taking 2GB, what Visual Studio reports here seems more like the normal memory usage that I should expect. It's not tracking the excess memory.

I'm going to take a semi-educated guess here, and say you were always attempting to use the 2.3GB.

There is no such thing as 2.3GB of user address space in 32 bit. The 32 bit address space is 4GB. Of that, Windows takes 2GB right off the top for itself and you can't touch it. So, right off the top, your available memory is 2GB. But here in the real world, Windows takes another so-many MB for the graphics card and such. So, if you're lucky, your program has about 1.6GB of memory, maximum. That's it. Not one byte more (not one byte more than your maximum I mean). Your numbers don't quite match up with that, but this is still the first thing that comes to mind for me.

Most of it seems to be Private memory which looks to be memory owned by that process that is not shared.

One way to test this theory is to turn on the /3GB flag at windows startup. This forces windows to use 1GB instead of 2GB. It starves Windows for memory that it desperately needs and thus this is a bad idea, but if your memory consumption on your app increases with this flag on, you will know it is not how the app is compiled but rather that 32 bit was hiding the problem. The /3GB flag will give your app a theoretical 3GB of memory space in 32 bit instead of a theoretical memory space of 2GB. I say theoretical, because like I said, Windows never gives you all the space it promises. None the less, you should see your 32 bit app start to consume more memory with that switch on. Turn it on for the test and then turn it back off once you know the answer. No harm, no foul.

If it consumes more memory under those conditions, I would say you have a memory leak of some type and 32 bit was preventing it from consuming any more because you consumed everything that was available and it ran out of memory before 2GB. The 64 bit address space is much larger, and so the app can consume what it wants with no 1.6GB limit. (Or maybe it's not a memory leak but just allocating everything it is allowed to. A memory leak suggests that you are not deallocating memory. But it could instead be that it is actually using the memory, but peaks out before it can get everything it wants in a 32 bit 2GB limit.)

This is just a guess, but it won't hurt to throw on that switch for a quick test, and then you'll know. The test would at least prove whether it is a problem with how it is compiled or whether it is a problem where being in a 32 bit environment hides the problem and the problem actually exists in both environments.

There's also a IMAGE_FILE_LARGE_ADDRESS_AWARE compiler flag, it looks like that should compile 32 bit to use 4GB on a 64 bit machine.

https://msdn.microsoft.com/en-us/library/windows/desktop/bb613473%28v=vs.85%29.aspx

You may be able to just change linker settings rather than the /3GB switch.

https://msdn.microsoft.com/en-us/library/wz223b1z.aspx

I did have a memory problem with a previous implementation where it crashed because it went over 1.6GB. The 3GB switch didn't work to work around the issue for some reason. Instead, I implemented a different design that reduced memory usage.

The 2.3GB is when running a complex script. If I run a basic test, I get 228MB in 32-bit and 381MB in 64-bit, so I don't think that will change anything.

You might try making it large address aware, like in that last link I sent. That's off by default for a 32 bit compile it looks like. Should be an easy quick test to rule one thing out.

I would expect a certain amount of expansion compiling in 64 bit. Your register sizes are twice as big. Memory addresses are twice as big, although admittedly 1.2GB of difference sounds a bit extreme. But that's why I lean towards thinking it could be a 32 bit wall.

Large Address Aware is one linker setting change. Should be easy to test. That should expand the address space to the full 4GB theoretically possible and let it consume more if it was previously hitting a 32 bit wall. Seems that's a bit less complicated than trying to get the /3GB switch to work in a 64 bit environment.

Just out of curiosity, I decided to take a look at my current OpenGL project, since I've got 32 bit and 64 bit versions in the build. When I go to Windows Task Manager with both of them running, The working set for the 32 bit version is 25MB and the working set for the 64 bit version is 30MB. The private working set is 14MB vs 18MB.

This is a program that does pretty much nothing but use OGL4.5 to draw a dancing triangle on the screen. I'm comparing release version to release version. Obviously, make sure you don't compare a debug version to a release version.

I decided to go a bit further and compare my most recent DX11 project between 32 bit and 64 bit builds. This program does a lot more work including animating actual 3D models, compared to the OGL program. Again, this is comparing release builds against each other.

The private working set on 32 bit vs 64 bit is 41MB for 32 bit and 37MB for 64 bit. Interesting that it's smaller for 64 bit. Commit size is 57MB for 32 bit and 337MB for 64 bit! That's a huge difference. Peak working set and working set are almost identical.

The difference in the Commit size is pretty substantial and it's the same code just a difference of whether it's compiled for 32 bit or 64 bit.

What's interesting is that in all cases, there are considerable memory allocation differences and I don't see what can be done about it.

.

You might try making it large address aware, like in that last link I sent. That's off by default for a 32 bit compile it looks like. Should be an easy quick test to rule one thing out.

Just tried that. It doesn't change anything.

Any other idea?