Archived

This topic is now archived and is closed to further replies.

Really big file I/O

This topic is 6362 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

While scribbling up a design for a general-purpose file class, I noticed that some of the operations I'm implementing often involve really large files (upwards of a few dozen megabytes). This got me wondering--in the ancient days of DOS, there were limits on file sizes using certain file I/O methods, and I'd freeze my PC frequently if I tried to work with really big files. (If you've ever worked with 90% of DOS hex editors, for example, you'll see what I mean.) So, anyway, I looked through my options, and I came out with three possibilities, though I'm probably missing many others: 1) FILE -- Ex:
FILE* infile = fopen("foo.dat","rb");   
2) fstream -- Ex:
fstream infile("foo.dat",ios::in | ios::binary);   
3) Platform SDK stuff -- Ex:
OFSTRUCT ofs; int filehandle = OpenFile("foo.dat",&ofs,OF_READ);  
What are the pros/cons of each method (or any method I've missed)? Do any of them have limits regarding file size? To my knowledge, #3 isn't portable to other operating systems, which might be a disadvantage depending on how resigned theprogrammer is to Windows' dominance in the OS arena. My main concern is that my program doesn't hang just because I'm reading/writing a really big movie file or something. Comments appreciated! SkyDruid Edited by - skydruid on 7/16/00 6:49:49 PM Edited by - skydruid on 7/16/00 6:51:01 PM

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
You also have the really low-level, unbuffered I/O stuff, like

    
int file;
file = _open("foo.bar", _O_BINARY);


then read/write using the _read/_write functions.
If you use this you have to do all buffering stuff by yourself.

I think (though I''m not completely sure) that all of the above alternatives use the same low-level routines implementaed in the windows kernel (or wherever it''s implemented)
So, id say your filesize limit is 4 gigabytes (32 bit filesystem) no matter which way you go (if your using windows and NTFS or FAT32 that is, I''m not sure about FAT16).

Share this post


Link to post
Share on other sites