Author
When you open a file using fopen, is the file actually being streamed from disk or loaded in to memory all at once? If streamed then, every read and write required disk access.
I'm trying to create a file i/o library and was thinking of loading the whole file into memory, allow user to manipulate it, and then save the file to disk if he or she wishes. But one problem with that is large files could very easily consume a large amount of memory.
Any suggestion?
The CRT buffers some of the file as required. My version of the CRT (MSVC 2003) buffers 4k at once. So when I open the file and read 1 byte, it reads 4k, and just returns the first byte. So the next 1023 bytes are effectively free to read.
After you read those 1023 bytes (or you do an fseek()), the OS needs to read from the disk again.
If you're only targetting Windows, you might want to look into memory mapped files. They allow you to just get a pointer to the first BYTE of the file, and the OS automagically pages parts of the file in and out of memory as needed.
There's similar functions for linux, but it means you have two codepaths.
After you read those 1023 bytes (or you do an fseek()), the OS needs to read from the disk again.
If you're only targetting Windows, you might want to look into memory mapped files. They allow you to just get a pointer to the first BYTE of the file, and the OS automagically pages parts of the file in and out of memory as needed.
There's similar functions for linux, but it means you have two codepaths.
It's technically implementation defined. However most implementations do buffered reads. For example, on the first read, it'll actually read say a kilobyte no matter how small the request is, and then subsequent reads will receive data from the buffer until the buffer is empty.
If you want to manipulate an entire file at once via memory, it's more reasonable to use memory mapped files. However, memory mapped files require platform specific functions.
If you want to manipulate an entire file at once via memory, it's more reasonable to use memory mapped files. However, memory mapped files require platform specific functions.
Author
Is there anyway to know the file size, without simply test reading every byte until the EOF and do byte count? Avoiding platform specify methods of course.
Quote:Original post by Simplicity
Is there anyway to know the file size, without simply test reading every byte until the EOF and do byte count? Avoiding platform specify methods of course.
fseek( pFile, 0, SEEK_END );unsigned int size = ftell();fseek( pFile, 0, SEEK_SET );
Some files don't have sizes, for they generate data as you read it. Finding the size of a file it platform-specific. You might get by, for ordinary files, by opening the file in binary mode, doing a seek() to the end of the file, and then a tell() to get the position. It is not guaranteed to indicate the file's size though (it is explicitely not when opening in text mode), but the odds are in your favour.
Note that it is possible to set the buffering mode in most CRT implementations using the setvbuf API.
Also, if you're using C++, boost::filesystem has a fairly portable file_size function that will tell you the size of a file.
Quote:Original post by FrunyWhat would seeking to the end of, say, /dev/random or /dev/null do, exactly?
Some files don't have sizes, for they generate data as you read it. Finding the size of a file it platform-specific.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
