Jump to content
  • Advertisement
Sign in to follow this  
caldiar

Ofstream Bails Out at 4096 KB

This topic is 3568 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Some of you may scoff at this but I'm trying to load more than 4096 KB of data into memory at once with ifstream and spit it back out with every 16 characters in reverse order in a resulting .raw file with ofstream. But! The .raw file stops being written to after it hits 4096 KB in size. Looking around some forum posts and on google it appears that the max buffer size is 4096 KB. I've been holding back from posting here because I didn't feel like this was really all that important but now it's nagging me in the back of my mind. Any other simple methods to read, reverse the data 8 bytes at a time, and write that you gurus can think of? I would really appreciate the input (no pun intended) :) Thanks! (I know, I'm horrible. No code. But I'm just looking for ideas)

Share this post


Link to post
Share on other sites
Advertisement
Have you tried flushing your buffers after a while? For example, you could read in 2048 kb, then flush your buffer, read another 2048 kb, flush your buffer, etc. Never mind, I'm completely wrong.

[Edited by - Moe on January 13, 2009 9:40:19 AM]

Share this post


Link to post
Share on other sites
You probably want to give some code or explain what calls you are using. Streams should flush automatically when their internal buffers exhaust - the buffering exists only for effeciency. In any case something like this works to reverse groups of 16 bytes (if I understand the problem) and doesnt appear to have any stream based limitation.

#include <fstream>
#include <iostream>
#include <vector>
#include <string>
#include <algorithm>

int main( int argc, char **argv)
{
std::vector< std::string> args( argv, argv + argc);

std::ifstream is( args.at( 1).c_str(), std::ios::binary);
is.seekg( 0, std::ios_base::end);
size_t sz = is.tellg();
is.seekg( 0, std::ios_base::beg);

std::vector< char> buf( sz);
is.read( &buf[ 0], buf.size());

int stride = 16;

for( int i = 0; i < buf.size(); i += stride)
std::reverse( buf.begin() + i, std::min( buf.begin() + i + stride, buf.end()));

std::ofstream os( ( args.at( 1) + ".rev" ).c_str(), std::ios::binary);
os.write( &buf[ 0], buf.size());
}

Share this post


Link to post
Share on other sites
Thanks guys :)

I played around and it was an obvious thing.

I wasn't reading with ios::binary and dumping it in to a buffer.

Once I did that and used some pointer arithmetic (took several tries to get the right things printing), my bytes are in reverse order now.

I didn't read the post about reverse until after I did my implementation which is ugly but serves its purpose as a personal utility for future image work.

For the curious readers:

I have a series of 4 bytes that are of interest to me in terms of reversing the order.

I set a pointer to the 11th position in the buffer array then save backwards in 4 2 char arrays (the bytes take up two chars in the .raw files I'm manipulating)

Then I just dump them out in reverse order.

It's slow, but it works.

Share this post


Link to post
Share on other sites
The buffers should flush properly irrespective of whether the approach used are calls (read/write) or stream iterators or overloaded stream operators (<<).

I initially misread your post and considered that you wanted to reverse the file as a whole, which dictated my code which dealt with it in memory in one go.

If the need is for rgba manipulation and if the files are really large (blue marble) you obviously need to avoid slurping it into memory, and must instead handle the IO in a more stream-like fashion. Otherwise for binary files of any kind, my approach at least, is to get it into memory (eg. a vector) as soon as possible since its more flexible and simple to deal with.

Share this post


Link to post
Share on other sites
I actually haven't determined whether the last 8 bits are being used or are there for padding.

The images are actually fairly small. For the purpose of my experiment I was loading a data block that had several images stored in one rather than splitting the block into individual chunks to view/save.

But yeah, I just wanted to go from the BGR format that the .raw file is saved in to the RGB format.

I need to take a look at my code again though because when I randomize the values of the bytes to get a multi-colored image and then try to read it back, I get a noisy olive green image.


Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!