Jump to content
  • Advertisement
Sign in to follow this  
Danicco

Binary Resource File Formatting

This topic is 2106 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm revisiting my Resource Class in my engine and I think it could use some improvements, so I'd like to ask for suggestions.

 

First, I'll explain what I have so far:

I got a class called ResourceCache which accepts a template class (of Textures, Meshes, AnimatedMeshes, etc) and keeps them in a vector for multiple instances of the same and resource management.

 

The class also have methods to add/remove, and to read/save from/to a file.

When saving, I pass a filename and pointer offset, and it'll call it's templates' own Save function. So Textures know how to Save/Load themselves, same as Meshes, AnimatedMeshes, Sounds, etc

 

The templates' Save function writes in binary mode in a format I decided, paired with it's Load, so I only keep the minimum relevant/necessary data for it.

 

The ResourceCache's Save function creates a header section so I don't have to read the whole file on every Load.

The header section is like this:

 

[long] headerStart
[long] headerEnd
 
//headerStart pointer - repeats this section till "headerEnd"
[int]  resourceID
[int]  resourceNameLength
[char] resourceName (* resourceNameLength)
[long] resourceDataPointer
//headerEnd pointer
 
//resourceDataPointer - repeats this till eof
//Varies according to resourceType
 
//File end

 

I can save multiple resource types (meshes, textures) in the same file with the Save function as well.

 

So, my concerns for now are:

 

1 - Currently I'm reading the header section for all resource files I type as I start the game, and saving them in a vector of "headers". This way when a resource is requested, I don't parse the read again, I just loop the vector, find the file pointer, move to it, and load the resource. I'm not sure how this method will work once my resource files get bigger and bigger. Any input on this?

 

2 - The format model I choose has an issue when developing, I have to manually (for now) insert new resources and save them in the binary files so I can use them (in the binary format), when I get my map editor this shouldn't be an issue but for now it's... really annoying and time consuming.

 

3 - The binary format I decided on doesn't seem a good idea if I ever need to add new resources, such as downloadable content and expansions. Since adding bytes in the middle of a file requires rewriting everything from that point forward (and I don't think I want to), I was thinking in separating between "header" file and "data" file, so if I ever need to add resources, it's just writing more bytes to the files' end.

 

Anyway, do you have any extra tips or suggestions, anything at all of what should I do that could improve this?

Share this post


Link to post
Share on other sites
Advertisement

Is there any reason you don't use an existing package format? I generally use regular zip files and use PhysFS to read them.

Share this post


Link to post
Share on other sites

No reason at all, I'm a beginner with C++ and engines and I never thought I could use it like that... that would solve 1 & 2, I'd need to look if I can add a new (downloaded) file to the main zip content or something like that.

 

On a side note, I've seen that the game Guild Wars 2 only have two files for the game, the executable and a single .dat file.

I've always wondered how they did that and I thought it was awesome to have the game folder look like that instead of a bunch of folders and files, but is that a good practice?

And since it's a MMO, there's always the regular updates and I'm still wondering how they handle that... I imagine having to rewrite the file (mine has 16.5gb now, I haven't updated in a while) isn't an option because of the size.

 

Any ideas on how I can achieve something like that? Or an advice if I shouldn't do it like that?

Share this post


Link to post
Share on other sites


No reason at all, I'm a beginner with C++ and engines and I never thought I could use it like that... that would solve 1 & 2, I'd need to look if I can add a new (downloaded) file to the main zip content or something like that.

I would strongly suggest for you to have a look into physfs too.

You can mount different *.zip and access the files of each through the same virtual Directory structure.

 

example zip #1:

/textures
    /walls
        wall_1.png

example zip #2:

/textures
    /walls
        wall_2.png

Your virtual directory structure will now look like this:

/textures
    /walls
        wall_1.png
        wall_2.png

If there exist several files with the same filename, the one inside the zip you added last will kind of "overwrite" (better "overlay") the ones in previous zips.

 

You can mix and match files which are just on your disk with files in zips. This has the huge advantage that you can store every file in directories during development, and just bundle these files into a zip prior to deploying your game.

 

 

Regarding Guild Wars 2:

This is just a wild guess but is based on several other games I've seen:

They might just be storing every asset in a plain directory tree on a disk and run some kind of "compiler" (or "packer) on it before a release. This "compiler" will gather all necessary files and might compress or preprocess some of them and outputs them into a game-specific container-file just like the one you created yourself.

The delivery of this file to all the PC's where it is installed will supposedly be done through a binary-diff where only changed data will be transferred (google can explain this better to you happy.png )

 

Regards

Markus

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!