Sign in to follow this  
MHOOO

Increasing fstream buffer for copy speedup?

Recommended Posts

MHOOO    148
Here's a piece of code that I got:
// l is the position in File1 at which we stop copying
File1.seekg(x,std::ios::beg); // x is the starting position
while(File1.good()) {
   if(File1.tellg()>=l)
      break;
   File2.put(File1.get());
}

Now this piece of code copies *incredibly* slow (its like 1MB/s). What this code is supposed to do, is to copy a set amount of data somewhere within File1 (not the whole file) to File2. Now I want to speed up things - I know that the speed issue is basically due to the fact that I have too many read/write accesses. I thought about rewriting the code to basically read complete chunks of File1 into memory with read/_Read_s() and write these chunks into File2 with write() but for something like that I'd have to rewrite & restructure the code. Now I've heard about basic_streambuf which basically is some sort of buffer as I understood it. Is it maybe possible to simply increase the buffering size of fstream in order for more input data & more output data to be stored in memory? In fact, I've tried the following already:
#define BUFFER_SIZE 1024*1024
char MyBuff1[BUFFER_SIZE],
     MyBuff2[BUFFER_SIZE];
File1.rdbuf()->pubsetbuf(MyBuff1,BUFFER_SIZE);
File2.rdbuf()->pubsetbuf(MyBuff2,BUFFER_SIZE);

// ... same as in 1st code example

Unfortunately this doesn't work as I get a Stack Overflow exception. 1.) Is it even possible to solve the problem (of speed) like this? 1a.) If it is, what am I doing wrong? 1b.) If it isn't, is there maybe a similar approach to solve the problem? Thanks & bye, MHOOO

Share this post


Link to post
Share on other sites
taby    1265
The easiest way to achieve this is to open your file using the std::ios::ios_base::binary parameter, and then use the read member function. Here you pass in the address of the target array, and how many chars to read from the file into it.

Using something like a 4K buffer makes a big difference.

Using a buffer that is the size of the entire file is even faster on MSVC++ .NET 2003.

Share this post


Link to post
Share on other sites
MHOOO    148
Well yes, that I already suggested myself, but its not exactly what I'd like to have at this point (as I'd have to rewrite my current code).
I'd rather like to know whether its possible to use std::basic_streambuf which is returned by file1.rdbuf() in order to change the used buffer.
On a sidenote - I do open the files with std::ios::binary.

Thanks & bye,
MHOOO

Share this post


Link to post
Share on other sites
Zahlman    1682
Yeah, use the read() interface. get() is for when you need to examine the individual characters as you read them. The fstream interface is basically a carbon-copy (updated to OOP) of the old C FILE* interface.

Oh, and you should know better than to use #defines for constants.

Anyway, you can see here for an example. (Please note that the example leaks the allocated memory; I hope you will be a bit more careful. It also just outputs the buffer to standard output; but you can instead write it to the other file just as easily; the ostream.write() interface is symmetric to istream.read().)

You can also do the same thing with iterative reads into a fixed-size buffer; each time you will want to read the minimum of the buffer size and the remaining amount of text, and write that amount out as well.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this