Jump to content

  • Log In with Google      Sign In   
  • Create Account

How to write to a small subset of a large file?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
7 replies to this topic

#1 Daban   Members   -  Reputation: 138

Like
0Likes
Like

Posted 04 March 2014 - 11:15 AM

Is it possible to write data to a custom place of a large file?

Suppose we have a big File Package which contains a lot of files and an end user adds or removes a file to/from it; now to update the package do we have to transfer all bytes of previous package to a new one (which will take a long to complete) or there is a better way?

 



Sponsor:

#2 Grimshaw   Members   -  Reputation: 657

Like
0Likes
Like

Posted 04 March 2014 - 11:39 AM

You can only rewrite subsets of a file like that if you put there the same amount of bytes. To be more clear, if you want to simply fit in there a bigger file than the subset size you're considering, it won't work.

 

If you want to replace a file in the package with another that is the same size , just rewrite that region with any File access API. If its smaller, you can rewrite it from byte 0 and leave the remaining space unused. If its bigger, the quickest way is to move all bytes that follow that region forward, and then fit the file in there.

 

I think the only other option is to just rebuild the entire package.


Indie Game Developer - Daeva Theory Studios

 

Twitter - Facebook

 

Next

Terra - rise of mankind

 

Previous

SFML Game Development (Book)


#3 Daban   Members   -  Reputation: 138

Like
0Likes
Like

Posted 04 March 2014 - 11:50 AM

thanx.



#4 nobodynews   Crossbones+   -  Reputation: 2118

Like
0Likes
Like

Posted 04 March 2014 - 01:31 PM

Does order the files are in matter? It might be acceptable to zero out the original file from the package, and then either append the file to the end of the package or search for a previously zeroe'd out hole in the middle of the package and put the file there. If you keep track of the wasted space you might automatically rebuild the package periodically if there's too much waste/gaps. Just a thought.

C++: A Dialog | C++0x Features: Part1 (lambdas, auto, static_assert) , Part 2 (rvalue references) , Part 3 (decltype) | Write Games | Fix Your Timestep!


#5 swiftcoder   Senior Moderators   -  Reputation: 10445

Like
0Likes
Like

Posted 04 March 2014 - 07:23 PM

The question in my mind would be whether or not you actually need to support random writes to a packaged archive. Filesystems are very well optimised for this case already, if you use a directory full of files instead of a packed archive.

 

In the case of games that use packed file archives, it is not uncommon to use an unpacked directory of files during development, and only produce the final packed archive when you are ready to release, thus avoiding the need to modify existing package files.


Tristam MacDonald - Software Engineer @Amazon - [swiftcoding]


#6 ankhd   Members   -  Reputation: 1367

Like
0Likes
Like

Posted 04 March 2014 - 07:54 PM

Hi.

you could write to a temp file all the stuff you want but the one you want to remove then add your new data to the temp and save the temp file .



#7 SeanMiddleditch   Members   -  Reputation: 7265

Like
1Likes
Like

Posted 04 March 2014 - 08:43 PM

Depending on your needs, it's often faster to just append the new file to the archive and then include metadata updates so readers know to grab the new/update version instead of any previous version in the archive and then repack the archive periodically. In general, any kind of I/O tends to do very well with sequential writes.

A further bonus is that appending is often safer (if you structure your metadata correctly) than updating data in the middle of an archive in the case your application crashes or is otherwise killed in the middle of a write, which your users will appreciate.

#8 Daban   Members   -  Reputation: 138

Like
0Likes
Like

Posted 05 March 2014 - 08:06 AM

so we can conclude that there is no magic way to do that. thank you so much for your help.






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS