Jump to content
  • Advertisement
Sign in to follow this  
DesignerX

Optimial file buffer size

This topic is 4888 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

If I want to copy a file (say about 700mb in size) to another file, using the fread, fwrite functions, what is the optimal buffer size used for this operation ?

Share this post


Link to post
Share on other sites
Advertisement
I'd say, the same as your disk's R/W cache, though a fraction of it would do OK too.

The only real solution is for you to try different buffer sizes, and see what gives the best result. Be curious. Experiment.

Share this post


Link to post
Share on other sites
However, be warned that what works well on one machine may not perform well on another due to different machine differences such as amount of ram and disk attributes.

With that being said if you've identified file coping as speed bottleneck of your overall application time you might want to look into using a memory memory mapped file copy if the system file copy or the fread/fwrite ones are not fast enough for you. The nice thing about memory mapped versions is that it eliminates the copy to and from the temporary buffer. In some cases that can spped things up alot.

Share this post


Link to post
Share on other sites
Quote:
Original post by Fruny
I'd say, the same as your disk's R/W cache, though a fraction of it would do OK too.


How would you actually determine this number, if you don't already know what exact hard drive is in the machine?

Share this post


Link to post
Share on other sites
You could do something really cool, like trying a range while copying the file, benchmarking each one then selecting that for the rest of the transfer [grin]

Share this post


Link to post
Share on other sites
That is question, what size to use that fits most of the machines of today and still give me good results ?

Share this post


Link to post
Share on other sites
The problem is that I don't think there's one good answer to your question. The "right" value can vary even on the same machine depending on whats running on it at a given time much less trying to coming up with a good value for all machines.

If this is something that's really critical for your application and you don't want to make it adaptive like paulecoyote suggested you might want to make it a user-configurable parameter. If you're using Windows use the cluster size of the disk as the default value. Look at GetDiskFreeSpace() and multiply lpSectorsPerCluster with lpBytesPerSector. This will get you the native granularity of the given disk.

I hope this helps.

Share this post


Link to post
Share on other sites
ASPI requests are or were limited to 64KiB, so best not to exceed that; 32KiB works well.
If you care about performance, use your platform's AIO mechanism instead. That bypasses any caching and queues commands to the drive so that throughput is maximized.

Share this post


Link to post
Share on other sites
I'd recommend a meg or so. And you should probably use a double-buffer (or triple-buffer) system with asynchronous access, if possible. That'd allow both reading and writing to occur simultaneously if they're not both on the same drive. A similar solution would be to use two threads: a reader and a writer thread, which alternate buffers. That would allow you to use synchronous access yet still overlap reads and writes.

Share this post


Link to post
Share on other sites
Noone has mentioned it yet, but just to state the (hopefully) obvious. The buffer size really should be a power of two!

I'd suggest 64K i.e. 65536 bytes. It most likely isn't going to perform much faster increasing it beyond that.

Oh and just remember that optimal can mean different things, so when you ask what is optimal, make sure you state what it is to be optimal in terms of, i.e. speed / memory usage etc...[smile]

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!