Sign in to follow this  

File I/O and speed

This topic is 4867 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

When loading from a file I've seen people do it two ways. I've seen them load the whole thing into a buffer, and then parse it from there, and I've seen them load it normally with read or whatever. What are the gains fo using the buffer method? I know any kind of IO can be slow, but is it a noticable gain by loading the whole fine in and parsing it that way? I've programmed for a long time, but it's funny, I've never profiled anything and I have no clue what's faster than something else except by basic algorithm analysis.

Share this post


Link to post
Share on other sites
Generally, it shouldn't matter much. Perhaps the reading in a buffer is faster, perhaps it's slower. I shouldn't bother about it. Just load everything into memory during the start of your application/game and show a "Please wait while we steal your personal data and transmit it to our server" screen.

And by loading into memory, I mean, do it just the way you like it. There shouldn't be much speed issues, except perhaps allocation the memory on the heap might take alot of time(Allocation memory is slow). And since you allocate memory twice(One for the raw data and one for the processed data, such as bitmap) and clean up once, I'd say loading into memory is a tad bit slower.

EDIT: HAR HAR! I beat Skizz by 3 seconds [grin]

Toolmaker

Share this post


Link to post
Share on other sites
Reading the whole file into a big buffer (or, at least, load big chunks) merely decreases the number of calls to the OS' file IO system. Calling the IO system adds a big overhead so doing it as little as possible will help. However, memory on modern OS's is virtual and could be paged, so, loading a big file could cause some other data to paged to disk which will slow things even further.
So, you're best off not worrying about it and just read files bit by bit. If it really is a problem then profile it to see if disk IO is what's slowing you down or if it's the processing of the loaded data.

Skizz

Share this post


Link to post
Share on other sites
Both your hard drive's hardware and your OS cache disk reads. When accessing the disk, the computer reads a whole block, not just a couple bytes. Look for information on your hard-drive cache size.

Reading data in large chunks does have the advantage that you aren't doing as many system calls and that your processing loop isn't interrupted.

Of course, if your file is too big to fully fit in memory in the first place, the decision is taken care of for you :) (though you can start playing with memory-mapped files...)

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
If reading a file from disc and do some parsing of the file on a Windows platform, I'd suggest using a memory-mapped file. Then the OS will load from disk as needed and performance for this is good as the OS is quite clever figuring out how to make this optimal.

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster
If reading a file from disc and do some parsing of the file on a Windows platform, I'd suggest using a memory-mapped file. Then the OS will load from disk as needed and performance for this is good as the OS is quite clever figuring out how to make this optimal.

Seconded. Especially if you're doing sequential access (not skipping around a lot). That will be the most efficient way to do it. Of course it only works on Windows, but theres probably a similar method on other systems. Worst case is that just end up doing 2 versions of the function and switching with a #define...

Share this post


Link to post
Share on other sites

This topic is 4867 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this