Sign in to follow this  
shirsoft

MSVC8 doesnt support pubsetbuf on stringstreams?

Recommended Posts

shirsoft    106
Hello All, Our code uses fstream to read/write floats one at a time so obviously it runs pretty slow. We were thinking of improving the performance without changing much of the code so decided to read the file into a istringstream and use this to extract the data. The problem is that if I pass the file data to the istringstream constructor, it truncates at the first null character. If I use pubsetbuf, it has no effect. Infact MSVC8 has a dummy implementation of setbuf. I wrote a small tester and it works fine on linux but not on MSVC8. Is there some workaround for this? Thanks, Shireesh
#include <sstream>
using namespace std;

int main()
{
	int n=321,n2=-1;
	istringstream ss((char*)&n,ios::binary);

	ss.rdbuf()->pubsetbuf((char*)&n,4);
	ss.read((char*)&n2,4);
	printf("%d\n",n2);
return 0;
}

Share this post


Link to post
Share on other sites
Brother Bob    10344
Since the std::istringstream constructor takes an std::string, not a char *, you can just take advantage of its constructors. More specifically, the one that takes a pointer and length.

char *buffer = new char[buffersize];
...
std::istringstream ss(std::string(buffer, buffersize), std::ios::binary);

Share this post


Link to post
Share on other sites
ApochPiQ    23003
Make sure you do some profiling to ensure that this change actually involves a speedup. On a modern OS with multiple layers of disk buffers chances are it isn't actually [significantly] faster to read in your entire file to memory before processing it; it's possible that all you really do is chew up a lot of RAM for not much gain.

I'm not saying that this is a foregone conclusion, mind you; again, that's why it's important to profile.

Share this post


Link to post
Share on other sites
shirsoft    106
Yes you are right, I didnt have any noticeable speed improvements. I guees the only way would be to read in bigger chunks I believe?

As for the RAM part, a typlical file is about 20mb containing mostly point data but we load it it ends up taking like 500mb of ram [embarrass]

Share this post


Link to post
Share on other sites
ApochPiQ    23003
I sincerely doubt you are going to get a speedup doing any kind of read operations on the file. There is a small chance you might get a boost from using memory-mapped files, but I wouldn't bet on it. If all you do is load the float data into a giant array and operate on that in memory, you might see some benefits from memory-mapped files, but if you do any kind of reorganization or post-processing that won't help you much.

Parsing out floats from a data stream just isn't very CPU intensive; you're going to run into a bottleneck at the stage of actually reading the file off the disk, and that bottleneck is going to be your limiting factor no matter how you load the file. (I'm assuming that using this data doesn't involve huge amounts of post-processing once it is loaded.) The OS is generally smart enough to minimize that bottleneck for you, so trying to outwit the OS probably won't yield any substantial benefits.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this