A problem with paging and threading.
Author
Hi,
my program allocates a lot of memory (a few hundred MB, tenthousands of allocs) with the main thread.
A different thread than allocates further memory and the system runs out of available system memory.
At this point a few hundred MB (!) are abruptly paged into the page file (I guess that these are the allocs from the main thread).
It appears that these pages are not written back to system memory if some allocs get freed.
I know that a thread context switch somehow causes paging, but that is as far as my knowledge about this stuff goes.
Has anyone a idea how I can prevent this violent paging?
Thanks a lot for your help!
Author
PS.:
I know that I might could try to use VirtualLock, but this would be too violent/hacky in my opinion.
I just dont want that this unnecessary paging happens.
I know that I might could try to use VirtualLock, but this would be too violent/hacky in my opinion.
I just dont want that this unnecessary paging happens.
How are you able to determine that the OS is doing those things?
If you use up all remaining RAM, then it's possible that the OS purges certain things from memory, in order to try and recover, to keep going.
The best way to prevent paging is to use less memory. What type of data are you using that is so large? If you're running out of memory, then clearly your program is using too much memory and this needs to be addressed, by means of compression and more space efficient structures etc.
If you use up all remaining RAM, then it's possible that the OS purges certain things from memory, in order to try and recover, to keep going.
The best way to prevent paging is to use less memory. What type of data are you using that is so large? If you're running out of memory, then clearly your program is using too much memory and this needs to be addressed, by means of compression and more space efficient structures etc.
Quote:Original post by Wanze
At this point a few hundred MB (!) are abruptly paged into the page file (I guess that these are the allocs from the main thread).
Standard windoze behaviour, the other problem is unability to switch threads in, on average, 1 ms interval.
Try to write few modifications of your application and try to isolate that problem a little, its often possible to decrease this problem. And fill a bug report to microsoft (That one about a time analisys of paging. In fact there might be some info on theirs site.)
Author
Quote:Original post by RagharDo you know about any online documentation about this?Quote:Original post by Wanze
At this point a few hundred MB (!) are abruptly paged into the page file (I guess that these are the allocs from the main thread).
Standard windoze behaviour
I can't find any good document on MSDN that explains the memory management for thread switches.
What I forgot to mention in my description is that my process instantly reduces it's working set although the total process allocation increased.
I use OpenThreads btw.
It goes like this:
main thread allocs-> Virtual Memory: ~800 MB Working Set: ~800 MB
... time goes on - process will run out of physical memory soon ...
child thread allocs-> Virtual Memory: ~1100 MB Working Set: ~200 MB(!!!WHY??)
I can't believe that this should be normal memory management behaviour.
This might be too low level, but maybe not: Memory Management: What Every Driver Writer Needs to Know
I don't think you're really gonna get that much paging in response to thread switching.
Check out Process Explorer to get an idea of just how much paging goes on compared with context switching. It's a free beefed up task manager.
[Edited by - LessBread on May 18, 2006 6:48:01 PM]
I don't think you're really gonna get that much paging in response to thread switching.
Check out Process Explorer to get an idea of just how much paging goes on compared with context switching. It's a free beefed up task manager.
[Edited by - LessBread on May 18, 2006 6:48:01 PM]
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement