How to pack your assests into one binary file (custom file format etc)

Started by
6 comments, last by Fredericvo 9 years, 9 months ago

I really know next to nothing about this topic, for example I have a commerical game here, it has one exe, one music.img and one data.img file and a couple of ini files etc, now these are not img files in the sense of mountable packed image files like iso, img, dmg rather I am assuming the developer just picked the extension img as a general file extension.

So these two files have all the assests stored in them, music, maps, textures, strings, etc, obioulsy in binary format. So how is this done, how to you package a bunch of file assests into one binary file like this, and why don't all devs do this, for example take a pk3 file which can obvioulsy be opened with winrar, so all the assests are available to see.

Any example, links or knowledge here?

Advertisement

How you do this is entirely up to the developer. There is no standard approach really....

On some occasions, it can be nothing more complicated than a zip or tar file.

Sometimes it may be a custom compressed file (probably using something like libZip to do the compression/decompression, and a custom header that will ensure it can't be opened as an archive).

Sometimes it may simply be a serialised memory dump containing all of the assets for a level.

And every so often, you may come across someone who uses encryption to add another layer of annoyance for people trying to rip the assets!

Whether or not a developer wants people to be able to navigate their asset packages, is going to depend largely on whether they care about protecting their art assets or not. For some games companies that support extensive modding in their games, they are likely to be much more open with their asset packages than a company that does not support modding. Ultimately there is no company wide standard governing this....

All you need is a file with a header section, an index section, and all the actual file data following. The index can be as simple as a list of pairs: file name and offset. Building the file, you take all your input files, and write them into a buffer while making index entries that record the name and offset. Then build your header and index, and dump the whole thing to disk.

(You can improve this process by writing placeholder header and index blocks to a file, then write the files directly to the target without an intermediary buffer. Then seek back to the beginning of the file and write the correct header and index.)

Reading is simple: load the index, use it to find the data. Maybe even memory map the whole thing.

You can choose the layer compression and more complex file arrangements on top of this scheme, but it's not really necessary. This is effectively how zip and tar files are laid out. Personally I just use a library like PhysicsFS instead of developing my own formats.

SlimDX | Ventspace Blog | Twitter | Diverse teams make better games. I am currently hiring capable C++ engine developers in Baltimore, MD.


... So how is this done, how to you package a bunch of file assests into one binary file like this ...

As Rob has mentioned, the solutions are numerous. The custom package file format I use has some header, followed by a table of content, followed by so-called load units, followed by streamable data.

The table of content has an entry for each single resource stored within the package. Each entry also stores the relative address and length of the load unit which contains the resource. A load unit may contain a single resource or a couple of resources. The latter is called a "bundle" and itself has a kind of table of content with back references to the package table of content. Bundles are fine for resources that should be loaded together (e.g. all the meshes and textures of a model). The resource is at least the description of what it is, but may also contain the resource data. If not, or if only partly, then the resource data or additional resource data is placed as streamable data towards the end of the package file. For example, textures may be stored with the low levels of detail embedded and the high levels of detail as streaming data elsewhere.

How about you put all of those into a zip using the zlib ? it has one of the best licenses over in the wild; and maybe it just solves exactly your packing problem, and provides compression as a cherry on top of the cake.

A windows-only solution is to use resources (.rc files) and access
then with FindResource(); LoadResource(); LockResource(); calls.

A windows-only solution is to use resources (.rc files) and access
then with FindResource(); LoadResource(); LockResource(); calls.

I don't think anyone would want to run a 4GB executable file, as resources can be huge.

Today's PC games are 64-bit and can have resources over 30 GB. But, my Internet connection hasn't changed, so I spent a week
just downloading it!

A windows-only solution is to use resources (.rc files) and access
then with FindResource(); LoadResource(); LockResource(); calls.


I don't think anyone would want to run a 4GB executable file, as resources can be huge.
Today's PC games are 64-bit and can have resources over 30 GB. But, my Internet connection hasn't changed, so I spent a week
just downloading it!

Ah I didn't know you were working on such a massive game. Resources can be spread over lots of dll's too. Some dll's are used for this very purpose and don't even have code.

This topic is closed to new replies.

Advertisement