Sign in to follow this  
Simplicity

FILE*

Recommended Posts

Simplicity    158
When you open a file using fopen, is the file actually being streamed from disk or loaded in to memory all at once? If streamed then, every read and write required disk access. I'm trying to create a file i/o library and was thinking of loading the whole file into memory, allow user to manipulate it, and then save the file to disk if he or she wishes. But one problem with that is large files could very easily consume a large amount of memory. Any suggestion?

Share this post


Link to post
Share on other sites
Evil Steve    2017
The CRT buffers some of the file as required. My version of the CRT (MSVC 2003) buffers 4k at once. So when I open the file and read 1 byte, it reads 4k, and just returns the first byte. So the next 1023 bytes are effectively free to read.
After you read those 1023 bytes (or you do an fseek()), the OS needs to read from the disk again.

If you're only targetting Windows, you might want to look into memory mapped files. They allow you to just get a pointer to the first BYTE of the file, and the OS automagically pages parts of the file in and out of memory as needed.
There's similar functions for linux, but it means you have two codepaths.

Share this post


Link to post
Share on other sites
SiCrane    11839
It's technically implementation defined. However most implementations do buffered reads. For example, on the first read, it'll actually read say a kilobyte no matter how small the request is, and then subsequent reads will receive data from the buffer until the buffer is empty.

If you want to manipulate an entire file at once via memory, it's more reasonable to use memory mapped files. However, memory mapped files require platform specific functions.

Share this post


Link to post
Share on other sites
Promit    13246
Quote:
Original post by Simplicity
Is there anyway to know the file size, without simply test reading every byte until the EOF and do byte count? Avoiding platform specify methods of course.
fseek( pFile, 0, SEEK_END );
unsigned int size = ftell();
fseek( pFile, 0, SEEK_SET );

Share this post


Link to post
Share on other sites
Fruny    1658
Some files don't have sizes, for they generate data as you read it. Finding the size of a file it platform-specific. You might get by, for ordinary files, by opening the file in binary mode, doing a seek() to the end of the file, and then a tell() to get the position. It is not guaranteed to indicate the file's size though (it is explicitely not when opening in text mode), but the odds are in your favour.

Share this post


Link to post
Share on other sites
SiCrane    11839
Also, if you're using C++, boost::filesystem has a fairly portable file_size function that will tell you the size of a file.

Share this post


Link to post
Share on other sites
Promit    13246
Quote:
Original post by Fruny
Some files don't have sizes, for they generate data as you read it. Finding the size of a file it platform-specific.
What would seeking to the end of, say, /dev/random or /dev/null do, exactly?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this