I have a small dilemma. I want to implement a scrambling technique for my file write methods. The saving method currently look like this: savedata(const void *data,uint size) { stream.write( (const char*) data, size ); } Every file input / output in my game will use this. The function is called per integer, per bool, etc (file.savedata(&hp_points,sizeof(int))). But to implement the scrambling, I need to do this: savedata(const void *data,uint size) { void *temp_data = new char[size]; memcpy( temp_data, data, size ); Scramble( temp_data, size ); stream.write( (const char*) temp_data, size ); delete [] temp_data; } Or I could do this: savedata(const void *data,uint size) { void *goes_out_exactly_like_it_came_in = (void*) data; Scramble( goes_out_exactly_like_it_came_in, size ); stream.write( (const char*) data, size ); Unscramble( goes_out_exactly_like_it_came_in, size ); } The allocation and deallocation of so much data will kill performance. I could write a resizable pool for the file object to allow temporary storage of the scrambled data. But it would add a lot of complication. I already know everyone will spit at me and tell me to obey the rules. So there's really no need to reply unless you believe that breaking the rules may not be that bad in this type of situation, or you have another way around this. Thanks much for any help.

void Scramble( void* output, const void* input, uint size );savedata(const void *data,uint size){    static std::vector<char> temp_buffer;    temp_buffer.resize( size );    void *goes_out_exactly_like_it_came_in = (void*) data;    Scramble( temp_buffer.begin(), goes_out_exactly_like_it_came_in, size );    stream.write( temp_buffer.begin(), size );}

where Scramble copies from the input buffer to the output buffer.

this way you allocate/deallocate a small finite number of times, (only as size grows toward its maximum value) since resize() doesn't deallocate.

Since it sounds like you're only calling it on smallish types, how about:

template <typename Data>void savedata(const Data &data) {    Data temp = data;    Scramble(temp, sizeof temp);    stream.write((const char*)&temp, sizeof temp);}

My C++ is rusty so that may not be correct, but you get the idea. Or, just do this:

template <typename Data>void savedata(Data data) {    Scramble(data, sizeof data);    stream.write((const char*)&data, sizeof data);}

[Edited by - Way Walker on July 20, 2006 10:20:12 AM]

DrEvil and AnonMike have the right ideas.

Quote:Original post by mikeman

Quote:The allocation and deallocation of so much data will kill performance.

I doubt that calling Unscramble would be any faster.

Why would you have any doubt at all?

I didn't put that into words very well. Allocating and deallocating thousands of tiny bits of data in Windows will completely destroy the performance to the point where you'll need a backup game to play while you wait for mine to load the next three yards of data into the map. Unscramble just performs simple value adjustments per byte. The difference isn't even worth comparing. Something like 1 to 80.

Quote:Original post by DrEvil
The allocation/de-allocation would be an insignificant performance impact compared to the actual File IO.

I think file IO is buffered in the background and flushed out in large chunks. Memory allocations are not buffered at all as far as I can tell. Allocations per member variable blows file IO performance hits out of the water. Saving a single small map area of my game results in about 3,000 ints being shoved out. That's 3,000 allocations and deletions per small area. The file IO buffer would probably jam that onto the hard drive in one single 12KB crunch.

Quote:Why don't you write out as you scramble, instead of putting it in a buffer? Then you only have to buffer whatever block size you scramble in? Do you scramble each byte? Each 4 bytes? etc... Then your scrambler can read in a block, scramble it, and write it straight out to the file. You don't need any temp buffers in this case, other than the byte or byte array(block size) that you scramble at once. No dynamic memory, just a fixed size local var in your scrambler. Allow your scrambler to take various outputs into the constructor, so you can have it scramble to a memory buffer, to a file, to cout, etc...

This is probably exactly what I should do. Instead of calling Scramble() if scrambling is enabled, I'll just diverge to that as the entire writing call instead of the normal call. Saving individual bytes to file at a time might hurt a bit, but nothing compared to most alternatives. I really hope fstream has an internal buffer.

Thanks for all of the advice.

Quote:Original post by Kest

Quote:Original post by mikeman

Quote:The allocation and deallocation of so much data will kill performance.

I doubt that calling Unscramble would be any faster.

Why would you have any doubt at all?

I didn't put that into words very well. Allocating and deallocating thousands of tiny bits of data in Windows will completely destroy the performance to the point where you'll need a backup game to play while you wait for mine to load the next three yards of data into the map.

Did you try it? How do you know how "thousands of tiny bits" get allocated and how (in)efficient it might be?

Even then, did you consider implementing an allocator to improve upon that?

Quote:Unscramble just performs simple value adjustments per byte. The difference isn't even worth comparing. Something like 1 to 80.

Then writing-encrypted-as-you-go should be even easier.

template<typename T>// btw, just say no to void* in C++void savedata(const T& thing) {  char* data = reinterpret_cast<char*>(&thing);  for (int i = 0; i < sizeof(T); ++i) {    stream.put(scramble_byte(data));  }};

Quote:

Quote:Why don't you write out as you scramble, instead of putting it in a buffer? Then you only have to buffer whatever block size you scramble in? Do you scramble each byte? Each 4 bytes? etc... Then your scrambler can read in a block, scramble it, and write it straight out to the file. You don't need any temp buffers in this case, other than the byte or byte array(block size) that you scramble at once. No dynamic memory, just a fixed size local var in your scrambler. Allow your scrambler to take various outputs into the constructor, so you can have it scramble to a memory buffer, to a file, to cout, etc...

This is probably exactly what I should do. Instead of calling Scramble() if scrambling is enabled, I'll just diverge to that as the entire writing call instead of the normal call. Saving individual bytes to file at a time might hurt a bit, but nothing compared to most alternatives. I really hope fstream has an internal buffer.

Keep scrambling separated out into a separate function. SRP, you know.

Also please note that with templates, you can avoid having to pass in the size value (as shown above). You gain type safety and a simpler interface.

Also please note that using this sort of technique to write basically anything that isn't a non-pointer primitive or pointer-less POD struct is Bad News(TM) (and even then...).

And yes, of course fstream has an internal buffer. But even if it didn't, it wouldn't be hard to implement your own buffering into a *static* buffer instead of allocating for every write. (Also have a look at the class used for fstream's buffer, std::streambuf.) It might actually be the best thing to implement a derived streambuf that 'scrambles' its contents when it is about to 'overflow', and swap that streambuf into the fstream when scrambling is desired.