How to write to a small subset of a large file?

Started by
6 comments, last by Daban 10 years, 1 month ago

Is it possible to write data to a custom place of a large file?

Suppose we have a big File Package which contains a lot of files and an end user adds or removes a file to/from it; now to update the package do we have to transfer all bytes of previous package to a new one (which will take a long to complete) or there is a better way?

Advertisement

You can only rewrite subsets of a file like that if you put there the same amount of bytes. To be more clear, if you want to simply fit in there a bigger file than the subset size you're considering, it won't work.

If you want to replace a file in the package with another that is the same size , just rewrite that region with any File access API. If its smaller, you can rewrite it from byte 0 and leave the remaining space unused. If its bigger, the quickest way is to move all bytes that follow that region forward, and then fit the file in there.

I think the only other option is to just rebuild the entire package.

thanx.

Does order the files are in matter? It might be acceptable to zero out the original file from the package, and then either append the file to the end of the package or search for a previously zeroe'd out hole in the middle of the package and put the file there. If you keep track of the wasted space you might automatically rebuild the package periodically if there's too much waste/gaps. Just a thought.

C++: A Dialog | C++0x Features: Part1 (lambdas, auto, static_assert) , Part 2 (rvalue references) , Part 3 (decltype) | Write Games | Fix Your Timestep!

The question in my mind would be whether or not you actually need to support random writes to a packaged archive. Filesystems are very well optimised for this case already, if you use a directory full of files instead of a packed archive.

In the case of games that use packed file archives, it is not uncommon to use an unpacked directory of files during development, and only produce the final packed archive when you are ready to release, thus avoiding the need to modify existing package files.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

Hi.

you could write to a temp file all the stuff you want but the one you want to remove then add your new data to the temp and save the temp file .

Depending on your needs, it's often faster to just append the new file to the archive and then include metadata updates so readers know to grab the new/update version instead of any previous version in the archive and then repack the archive periodically. In general, any kind of I/O tends to do very well with sequential writes.

A further bonus is that appending is often safer (if you structure your metadata correctly) than updating data in the middle of an archive in the case your application crashes or is otherwise killed in the middle of a write, which your users will appreciate.

Sean Middleditch – Game Systems Engineer – Join my team!

so we can conclude that there is no magic way to do that. thank you so much for your help.

This topic is closed to new replies.

Advertisement