Sign in to follow this  
MuTeD

[C++] Opening large files, and mmap

Recommended Posts

Hello, I wrote this sub-routine (not to be confused with ReadFile() function from the windows.h header): void readFile(char Filename[]) { fstream FileMod; MEMORY_BUFFER = new char[BUFFERED]; // allocate memory FileCharacter = new char[BUFFERED]; // allocate memory FileMod.open(Filename, ios_base::in); // Copy contents of file X into string ReadInto while ( (*FileCharacter = FileMod.peek()) != EOF ) { FileMod.getline(MEMORY_BUFFER, 1000000, 0x0E); ReadInto = MEMORY_BUFFER; // convert from *CHAR (*MEMORY_BUFFER) to STRING (ReadInto) } FileMod.close(); delete(MEMORY_BUFFER); // release the allocated memory delete(FileCharacter); // release the allocated memory return; } It can open any file in about 2 seconds (human time), highest I've tested was with 200 MB+ file. I've checked on the 'net (google of course), and read things about the "mmap system," My question was, is how I open the file faster/as effecient as the Memory Mapping way?

Share this post


Link to post
Share on other sites
Quote:

My question was, is how I open the file faster/as effecient as the Memory Mapping way?
You just don't, with that approach.

mmap's great advantage is that it lets the operating system's vm handle the nasty details of efficient page replacement for you, and doesn't actually do much more io than it needs to, unlike your approach.

That's why you can efficiently handle a 200 gb file using mmap and posix madvise on a conventional 64 bit desktop system, and why your approach to 'opening' that same file would bring your machine to a grinding halt, no matter how fast your streaming getline function is.

So a better question to ask yourself might be: do you understand that using a file doesn't necessarily mean loading all of it into physical memory at once?

If you do, and need mmap's efficiency, why are you not using mmap?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this