Sign in to follow this  
Wanze

A problem with paging and threading.

Recommended Posts

Hi, my program allocates a lot of memory (a few hundred MB, tenthousands of allocs) with the main thread. A different thread than allocates further memory and the system runs out of available system memory. At this point a few hundred MB (!) are abruptly paged into the page file (I guess that these are the allocs from the main thread). It appears that these pages are not written back to system memory if some allocs get freed. I know that a thread context switch somehow causes paging, but that is as far as my knowledge about this stuff goes. Has anyone a idea how I can prevent this violent paging? Thanks a lot for your help!

Share this post


Link to post
Share on other sites
PS.:

I know that I might could try to use VirtualLock, but this would be too violent/hacky in my opinion.
I just dont want that this unnecessary paging happens.

Share this post


Link to post
Share on other sites
How are you able to determine that the OS is doing those things?
If you use up all remaining RAM, then it's possible that the OS purges certain things from memory, in order to try and recover, to keep going.

The best way to prevent paging is to use less memory. What type of data are you using that is so large? If you're running out of memory, then clearly your program is using too much memory and this needs to be addressed, by means of compression and more space efficient structures etc.

Share this post


Link to post
Share on other sites
Quote:
Original post by Wanze
At this point a few hundred MB (!) are abruptly paged into the page file (I guess that these are the allocs from the main thread).

Standard windoze behaviour, the other problem is unability to switch threads in, on average, 1 ms interval.
Try to write few modifications of your application and try to isolate that problem a little, its often possible to decrease this problem. And fill a bug report to microsoft (That one about a time analisys of paging. In fact there might be some info on theirs site.)

Share this post


Link to post
Share on other sites
Quote:
Original post by Raghar
Quote:
Original post by Wanze
At this point a few hundred MB (!) are abruptly paged into the page file (I guess that these are the allocs from the main thread).

Standard windoze behaviour
Do you know about any online documentation about this?
I can't find any good document on MSDN that explains the memory management for thread switches.


What I forgot to mention in my description is that my process instantly reduces it's working set although the total process allocation increased.
I use OpenThreads btw.


It goes like this:

main thread allocs-> Virtual Memory: ~800 MB Working Set: ~800 MB
... time goes on - process will run out of physical memory soon ...
child thread allocs-> Virtual Memory: ~1100 MB Working Set: ~200 MB(!!!WHY??)

I can't believe that this should be normal memory management behaviour.

Share this post


Link to post
Share on other sites
This might be too low level, but maybe not: Memory Management: What Every Driver Writer Needs to Know

I don't think you're really gonna get that much paging in response to thread switching.

Check out Process Explorer to get an idea of just how much paging goes on compared with context switching. It's a free beefed up task manager.


[Edited by - LessBread on May 18, 2006 6:48:01 PM]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this