Sign in to follow this  

[C++] Opening large files, and mmap

This topic is 3374 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello, I wrote this sub-routine (not to be confused with ReadFile() function from the windows.h header): void readFile(char Filename[]) { fstream FileMod; MEMORY_BUFFER = new char[BUFFERED]; // allocate memory FileCharacter = new char[BUFFERED]; // allocate memory FileMod.open(Filename, ios_base::in); // Copy contents of file X into string ReadInto while ( (*FileCharacter = FileMod.peek()) != EOF ) { FileMod.getline(MEMORY_BUFFER, 1000000, 0x0E); ReadInto = MEMORY_BUFFER; // convert from *CHAR (*MEMORY_BUFFER) to STRING (ReadInto) } FileMod.close(); delete(MEMORY_BUFFER); // release the allocated memory delete(FileCharacter); // release the allocated memory return; } It can open any file in about 2 seconds (human time), highest I've tested was with 200 MB+ file. I've checked on the 'net (google of course), and read things about the "mmap system," My question was, is how I open the file faster/as effecient as the Memory Mapping way?

Share this post


Link to post
Share on other sites
Quote:

My question was, is how I open the file faster/as effecient as the Memory Mapping way?
You just don't, with that approach.

mmap's great advantage is that it lets the operating system's vm handle the nasty details of efficient page replacement for you, and doesn't actually do much more io than it needs to, unlike your approach.

That's why you can efficiently handle a 200 gb file using mmap and posix madvise on a conventional 64 bit desktop system, and why your approach to 'opening' that same file would bring your machine to a grinding halt, no matter how fast your streaming getline function is.

So a better question to ask yourself might be: do you understand that using a file doesn't necessarily mean loading all of it into physical memory at once?

If you do, and need mmap's efficiency, why are you not using mmap?

Share this post


Link to post
Share on other sites

This topic is 3374 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this