Jump to content
  • Advertisement

Archived

This topic is now archived and is closed to further replies.

Stark

Packed files

This topic is 6817 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Well, the easiest way is just to include the directory in the file header for a given file. For all the files in the sound directory, their file name stored in the package would read "sounds/????????.???"

Now, if you want to get tricky, you use recursive package files. As in, the sounds directory is really a package stored in the master package file. This speeds stuff up tremendously when accessing files, period.

My package format supports/uses both.

- Splat

Share this post


Link to post
Share on other sites
Advertisement
Neat idea, Splat. So you header conatins an entry, say, sounds that actually points to a nested header with info about all of the sounds info?

I never thought of that. hmmm.... Gonna give it a try. Thanks.

By the way, anybody here used zLib? Anygood? I'm not sure if I should use it, as it seems to be a little overkill.

Share this post


Link to post
Share on other sites
Yeah, pretty much. The "sounds" entry could actually be extracted from the main package and would yield a completely valid archive on its own. Quite cool. The packing utility took a little tinkering before this feature worked as intended, but its cool now. The packer is almost ALWAYS more complex than the depacker (in-game).

- Splat

[This message has been edited by Splat (edited November 17, 1999).]

Share this post


Link to post
Share on other sites
I agree with Zenroth. I've found it relatively easy to pack the data. It's extracting it properly that seems to be more complicated.

Share this post


Link to post
Share on other sites
Are you using recursive packages? With me, the loading process is VERY straightforward. First, the directory of the main package is loaded, which contains file headers containing the filename, size, offset into the current package, and compression format.

Then, for each subpackage, the process is repeated, creating a tree of file headers.

Whenever my game needs a file, it passes the complete filename it wants, ie "sounds/death.wav". First, it checks to see if a file header with that exact name is present. If not, it "peels" off the directory and looks for a file header of a subpackage with that name. Recurse the same function into the subpackage. Repeat until file is found or conclusively not there.

Once the file is found, we have (automagically using the loader function) an absolute offset from the beginning of the WHOLE package. Its just a matter of reading it in.

The packer, however, is a real recursive nightmare. Reason being, each file header is NOT the same size. The file name has a variable size and is stored by writing a byte for how long it is then the actual contents. So to pack the archive, you need to reserve the file header space at the beginning, then compress the files, then come back and record the file header's with offset, size, and compression type.

Its much more important to have a simple loader than a simple packer. You have unlimited time to pack the data, but a game user does NOT want to wait a minute while your game loads its data. Redesigning a file format for easy loading is critical.

- Splat

[This message has been edited by Splat (edited November 17, 1999).]

Share this post


Link to post
Share on other sites
That is very interesting. Thanks for the clarification. The only question I have left is, why do you use a variable size header, especially if you're concerned about speed? Wouldn't it be faster to use a static size?

Share this post


Link to post
Share on other sites
Well, here's the thing: I am concerned about speed, but only on the loading end. While it takes a little longer to pack, because you have to traverse the file list twice in effect. (note: I may be fixing this by placing the file header at the end of a package rather than the beginning: this also speeds up additions and removals to the package without a need to completely repack it)

However, reading a byte then a variable number has neglible effect on performance, and that is all the loader does. The trick is, loading of the package directory is done at registration (in my game engine you register a package file to get an integer package id). So that is also just a one shot thing. Plus, it keeps support for long filenames while not having to reserve 255 bytes per entry. You got several thousand files, that 255 bytes per will start to add up and make the file header so very large.

I guess in the end its just a personal choice. 400k here or there isn't that big a deal.

- Splat

Share this post


Link to post
Share on other sites
Ok. SO one last question.

How do you deal with compression? DO you compress the packed file, or each file before packing it? I see advantages both ways:

one big archive - Less load time, decompress at startup. After that, just extract data. More memory is required to hold the decompressed file.

bunch of archives - less memory. Only unpacked as they're needed. Slow because they're only unpacked as they're needed.

So, what do you think? Compress before or after packing, or just skip packing altogether. Although, if I'm gonna skip it, I might as well just compress it before and after startup.

Share this post


Link to post
Share on other sites
Did any of you guys consider not re-inventing the wheel and use ZIP? Just wondering if there's something important I missed

/Niels

Share this post


Link to post
Share on other sites

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!