# Binary Resource File Formatting

This topic is 2106 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I'm revisiting my Resource Class in my engine and I think it could use some improvements, so I'd like to ask for suggestions.

First, I'll explain what I have so far:

I got a class called ResourceCache which accepts a template class (of Textures, Meshes, AnimatedMeshes, etc) and keeps them in a vector for multiple instances of the same and resource management.

The class also have methods to add/remove, and to read/save from/to a file.

When saving, I pass a filename and pointer offset, and it'll call it's templates' own Save function. So Textures know how to Save/Load themselves, same as Meshes, AnimatedMeshes, Sounds, etc

The templates' Save function writes in binary mode in a format I decided, paired with it's Load, so I only keep the minimum relevant/necessary data for it.

The ResourceCache's Save function creates a header section so I don't have to read the whole file on every Load.

The header section is like this:

[long] headerStart

[int]  resourceID
[int]  resourceNameLength
[char] resourceName (* resourceNameLength)
[long] resourceDataPointer

//resourceDataPointer - repeats this till eof
//Varies according to resourceType

//File end


I can save multiple resource types (meshes, textures) in the same file with the Save function as well.

So, my concerns for now are:

1 - Currently I'm reading the header section for all resource files I type as I start the game, and saving them in a vector of "headers". This way when a resource is requested, I don't parse the read again, I just loop the vector, find the file pointer, move to it, and load the resource. I'm not sure how this method will work once my resource files get bigger and bigger. Any input on this?

2 - The format model I choose has an issue when developing, I have to manually (for now) insert new resources and save them in the binary files so I can use them (in the binary format), when I get my map editor this shouldn't be an issue but for now it's... really annoying and time consuming.

3 - The binary format I decided on doesn't seem a good idea if I ever need to add new resources, such as downloadable content and expansions. Since adding bytes in the middle of a file requires rewriting everything from that point forward (and I don't think I want to), I was thinking in separating between "header" file and "data" file, so if I ever need to add resources, it's just writing more bytes to the files' end.

Anyway, do you have any extra tips or suggestions, anything at all of what should I do that could improve this?

##### Share on other sites

Is there any reason you don't use an existing package format? I generally use regular zip files and use PhysFS to read them.

##### Share on other sites

No reason at all, I'm a beginner with C++ and engines and I never thought I could use it like that... that would solve 1 & 2, I'd need to look if I can add a new (downloaded) file to the main zip content or something like that.

On a side note, I've seen that the game Guild Wars 2 only have two files for the game, the executable and a single .dat file.

I've always wondered how they did that and I thought it was awesome to have the game folder look like that instead of a bunch of folders and files, but is that a good practice?

And since it's a MMO, there's always the regular updates and I'm still wondering how they handle that... I imagine having to rewrite the file (mine has 16.5gb now, I haven't updated in a while) isn't an option because of the size.

Any ideas on how I can achieve something like that? Or an advice if I shouldn't do it like that?

##### Share on other sites

No reason at all, I'm a beginner with C++ and engines and I never thought I could use it like that... that would solve 1 & 2, I'd need to look if I can add a new (downloaded) file to the main zip content or something like that.

I would strongly suggest for you to have a look into physfs too.

You can mount different *.zip and access the files of each through the same virtual Directory structure.

example zip #1:

/textures
/walls
wall_1.png

example zip #2:

/textures
/walls
wall_2.png

Your virtual directory structure will now look like this:

/textures
/walls
wall_1.png
wall_2.png

If there exist several files with the same filename, the one inside the zip you added last will kind of "overwrite" (better "overlay") the ones in previous zips.

You can mix and match files which are just on your disk with files in zips. This has the huge advantage that you can store every file in directories during development, and just bundle these files into a zip prior to deploying your game.

Regarding Guild Wars 2:

This is just a wild guess but is based on several other games I've seen:

They might just be storing every asset in a plain directory tree on a disk and run some kind of "compiler" (or "packer) on it before a release. This "compiler" will gather all necessary files and might compress or preprocess some of them and outputs them into a game-specific container-file just like the one you created yourself.

The delivery of this file to all the PC's where it is installed will supposedly be done through a binary-diff where only changed data will be transferred (google can explain this better to you  )

Regards

Markus

• 21
• 13
• 9
• 17
• 13