• Advertisement
Sign in to follow this  

How do game engineers pack their data?

This topic is 1080 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

In order to import 3d models, a 3d model file (i.e. COLLADA) and some maps are needed.

Having all that data out there in some folder isn't too smart because it can be viewed or even worse, edited, easily by anyone, so I noticed that many games have their data packed in one or many large files.

How do game engineers do that? Any thoughts on this matter?

Share this post


Link to post
Share on other sites
Advertisement

In general, people will tinker with and change the assets for your game if they love it.

So, even if you pack and compress or align your game data, consider making a small sdk with the tools so that people don't have to be ingenious to get at it. :)

Those tools includes pack readers, model viewers, etc.

Share this post


Link to post
Share on other sites

This is a nice article on prepping data offline ready for efficient loading: http://www.gamasutra.com/view/feature/132376/delicious_data_baking.php?page=1

 

Another thing to think about is whether you want to use a 'push' or a 'pull' approach to loading resources.

 

The pull approach is perhaps more natural, you have a bunch of resources set out into a logical (to a human) folder structure (possibly in some sort of zip-esque archive format), then your game code requests the assets it needs in some arbitrary order and they get loaded. Sometimes when you're loading a model, it might contain dependencies on some texture or sound, so you fire off loads for them too.

 

The problem with the pull approach is that you're seeking all over the place. Which is fine for solid state storage (SSD, cartridges, mobile) is OK for hard disk drives, but is a total nightmare for DVD/Blu-Ray, etc, as the seek costs can be huge.

 

The push model is an approach where you create a single file for a collection of stuff that will get loaded together. So if your game is level based and non-streaming, your tools might construct a single level file containing all the appropriate models, sounds and textures. The game code will request to load the level file, and the load process will push each of the resources it comes across into the right subsystem.

 

I guess with the current crop of platforms, and the fact that games stream so much dynamically, the push vs pull choice is less important, and it's always possible to make the pull model work OK by carefully laying out resources on the storage media, but I think the push approach is the better one if you're planning out a system from scratch.

Share this post


Link to post
Share on other sites

If you're using C or c++, look into physicsfs.

 

It allows loading resources from compressed archive formats such as zip, 7z, and wad. It acts like a filesystem so you can stream files from it or just load them whole into ram. Definitely worth the effort to integrate. 

Share this post


Link to post
Share on other sites

Firstly, you probably don't want an interchange format like COLLADA or the native format of your modelling package in your final game; it has all the data needed for editing your model, and a lot of that simply isn't needed in the final game, so you'll usually want an optimized format that only contains the information needed by your game, preferably in a format that can be loaded and used as-is without or with very minimal further processing.

 

Can you name file formats that match this "optimized format" description?

 

One more question. Let's say I have this one big .dat file in my game directory where data is present sequentially, and I load it.

Will I need to have tons of "#define X_ASSET_OFFSET" (and perhaps X_ASSET_SIZE) and then call ifstream::seekg() when I load every model?

Share this post


Link to post
Share on other sites


Can you name file formats that match this "optimized format" description?

They are usually specific to individual games, or at least game engines.

 

Generalising such a format to be an open standard would sort of defeat the purpose of a highly-specific format optimised to the needs of the individual game.

Share this post


Link to post
Share on other sites

One more question. Let's say I have this one big .dat file in my game directory where data is present sequentially, and I load it.
Will I need to have tons of "#define X_ASSET_OFFSET" (and perhaps X_ASSET_SIZE) and then call ifstream::seekg() when I load every model?

 

Actual Implementation is pretty specific to platform/library, but you definately want to avoid manually specifying offsets. Usually on Initialize you walk the files content and generate a content map (resource id with file offset). Then when you want to load a given resource, you lookup the id and read from that offset. (Map Generation could potentially be something performed offline).

Edited by theflamingskunk

Share this post


Link to post
Share on other sites

 


Can you name file formats that match this "optimized format" description?

They are usually specific to individual games, or at least game engines.

 

Generalising such a format to be an open standard would sort of defeat the purpose of a highly-specific format optimised to the needs of the individual game.

 

Then there preferable format should be some kind of binary format (unlike collada, that just points to the assets on the disk) that encapsulates the assets' binaries? Am I grasping this wrong?

Share this post


Link to post
Share on other sites


Then there preferable format should be some kind of binary format (unlike collada, that just points to the assets on the disk) that encapsulates the assets' binaries? Am I grasping this wrong?

Yes.

 

Theory runs that a packed file format contains 'chunks' of raw data (vertex data, audio buffers, etc) which are laid out exactly as they will be needed in memory. Loading these chunks is a straightforward read/mmap operation, with no further processing required.

 

In order to know what chunks you need to read/mmap, you also need metadata (basically, an index to the packed file). These are stored in their own chunks, which you read in, process, and then use to load the remaining chunks. The metadata chunks should generally be very small compared to the data chunks, so these are not always stored in binary - I've seen systems that store metadata in JSON.

Share this post


Link to post
Share on other sites

In your theory, the assets of each level should be stored in a different file?

What if there's this AK47 model that can be displayed throughout the entire game? Should there be a "general" data file that contains these kind of assets?

I'm sorry but I want to fully understand this subject. A link to a nice article on this will suffice too if you don't feel like answering again.

Share this post


Link to post
Share on other sites


In your theory, the assets of each level should be stored in a different file?

Some systems place the entire game assets in a single packed file. Some use common packs, plus packs for sets of levels.

 

Still others can load levels either from packed assets, or individual files. Many of blizzard's games work this way - the campaign levels are all packed into the main archive, but add-on multiplayer levels are downloaded and stored individually...

Share this post


Link to post
Share on other sites

In your theory, the assets of each level should be stored in a different file?

 

They do not need to be.

 

The actual structure you go for is very dependent upon the design of the game. It may make sense to have a set of common packages that are shared for all the code, and then individual packages that are only used for certain levels (so levels that dont use that package dont care about it).

 

You might have each package (binary data file with set of assets) store only one type [so common_audio.pkg only contains audio data, common_mesh.pkg only contains mesh data] or you might store it logically so that a package has many different types that are used together [weapons.pkg that contains mesh, audio, texture and material data for your weapons].

Edited by theflamingskunk

Share this post


Link to post
Share on other sites

So I have been wondering the same thing for a while now, and I have few questions.

 

So lets say you compress all you game assists into a single zip file format. Doesn't it take longer to decompress the zip file into your hard drive and then read all its contents into the game? Or do you not decompress it to the hard drive? Is there a way to read the game assists without decompressing it to the hard drive? How does it really work?

 

[Update]

Well that was a stupid question. I totally forgot that you can load files into a MemoryStream. So you would just load the zip file contents into a MemoryStream without having to decompress it to the hard drive. 

Edited by FantasyVII

Share this post


Link to post
Share on other sites


Well that was a stupid question. I totally forgot that you can load files into a MemoryStream. So you would just load the zip file contents into a MemoryStream without having to decompress it to the hard drive. 
Exactly, the whole point of adding compression is that sometimes reading uncompressed data takes longer than reading it compressed into memory and uncompress it on the fly.

 

Although ZIP is kinda heavy weight for decompression AFAIK, you could try with other faster methods like LZ4 (that dont give such good compression rations though). As with anything, you'd need to test it out.

Share this post


Link to post
Share on other sites

 


Then there preferable format should be some kind of binary format (unlike collada, that just points to the assets on the disk) that encapsulates the assets' binaries? Am I grasping this wrong?

Yes.

 

Theory runs that a packed file format contains 'chunks' of raw data (vertex data, audio buffers, etc) which are laid out exactly as they will be needed in memory. Loading these chunks is a straightforward read/mmap operation, with no further processing required.

 

In order to know what chunks you need to read/mmap, you also need metadata (basically, an index to the packed file). These are stored in their own chunks, which you read in, process, and then use to load the remaining chunks. The metadata chunks should generally be very small compared to the data chunks, so these are not always stored in binary - I've seen systems that store metadata in JSON.

 

Sorry for bumping. What 3d file formats do you recommend then?

Share this post


Link to post
Share on other sites


What 3d file formats do you recommend then?

For exchanging data between content creation tools and your art pipeline? Any of a number of interchange formats, such as COLLADA or FBX.

 

For use at runtime? The custom file format which you are going to write to support the exact needs of your particular game.

Share this post


Link to post
Share on other sites

I've been using OpenGEX format, there are export plugins for all major 3D editors, and one C++ import implementation, all in that page. There is also a Java OpenGEX importer, which is the one I am using since its self contained (ie, no need to use half of libGDX or jME like for other formats).

 

As other people said, its not a good idea to use text based formats for runtime though. Best to code up a tool that imports whatever exchange format you use and spits out a binary blob that your engine can easily load. 

Edited by TheChubu

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement