• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
Alundra

Package File Format

10 posts in this topic

Hi all,

Package is a need because you can have a faster load time because you open a big file and not a lot of files.

When you works on content, you works without package and then you pack the folder at the end.

When you pack, you split each 4go, I think this number is not bad to have nice package size.

Now the question is how makes a good package file format and what compression use.

Is ZLIB the best choice to compress data for a package file format ?

Is a good package file format only :

- Header
- File table
- File data

Since the content on disk is a folder based, the file table should contains the full path using this folder has root.

Is it better to store like that or have a hierarchical file table ?

Thanks

Edited by Alundra
0

Share this post


Link to post
Share on other sites
This means that you can't have two assets with the same name, but I see this as a positive feature rather than a negative ;-)

How can you differentiate if it has the same name ?

In UnrealEngine you can't have two asset using the same name, maybe they flat all using name only.

EDIT : I just tried and you can have the same name if not in same folder, so surely a flat hierarchy name.

 

The last engine that I used, used paths within filenames (e.g. An asset might be called "foo/bar/baz.type.platform") - so just a flat/non-hierarchical table containing these long names.

Looks not bad, that avoid recursion to search file data.

 

During development, assets are stored as individual files (not packaged) so it's easy to make changes to the game.

This is better yea, Unity works like that, Unreal Engine changed on his version 4 using this method as well.

A Lot of user of unreal engine was not happy to always works using package all the time.

 

We used to mount a file as a file system, it gave you massive speed increases on windows.
We used to call it a Virtual File System, here is an example.

Is it really a need ?

 

-----

 

What's about the compression ? That still a question.

Edited by Alundra
0

Share this post


Link to post
Share on other sites

 

What's about the compression ? That still a question.

 

 

You can build that in if you want to. Zlib is pretty fast, but it's not as fast as just loading a file so you will have to make a judgement call to see if it is worthwhile.

 

I use zipped file systems all the time for mobile devices, but it's not always a good idea.

0

Share this post


Link to post
Share on other sites
Once this is done the users of the assets can load assets individually by asset ID, which basically just amounts to a binary search through the map and then returning a pointer once the asset is found.

Do you keep loaded the file or use fopen when you want to keep a file data ?

Is it bad to fopen/fclose for each file in a GetMemoryFile( const int EntryIndex ) ?

Edited by Alundra
0

Share this post


Link to post
Share on other sites
Same as MJP here but we don't store flat list hashes, we either use a hash for full path or hash per directory/file depending on complexity and use a minimal perfect hash for lookup.
0

Share this post


Link to post
Share on other sites

back in the day, they were called a WAD file or resource file. used to do them myself for GAMMA Wing (circa 1995).  i did an in-house implementation that supported all the basic file formats used by the company's titles, and included things like on-the fly decompression from the resource file into ram. one unusual feature was separate resource and index files. the index file was opened, it was used to read all resources at program start from the resource file, then both files were closed.

 

nowadays, i keep everything out in the open for easy modding by fans.

0

Share this post


Link to post
Share on other sites

Using zlib or LZMA SDK is pretty common. Use it to compress each individual file, [...] Unless the user has an SSD or RAM-disk, this should actually be a lot faster than loading uncompressed files!

This is very true, but the fact that it is commonly done does not mean it is necessarily correct. My guess is that a lot of people simply use ZLib "because it works", and because it has been used in some well-known virtual filesystem libraries where it has proven to work, too. That was at a time when the goal of compression was slightly different, too (reduce storage space).

 

I wouldn dearly recommend LZ4/HC over ZLib.

 

You compress to gain speed, not to save space. This is rather obvious, but it should still be re-iterated, so there is no misunderstanding. Disk space is abundant and cheap, and hardly anyone will notice half a gigabyte more or less, but long load times suck.

 

You gain speed if, and only if, loading the compressed data and decompressing is faster than just loading raw data. Again, this is actually pretty obvious.

 

ZLib decompression speed is around 60 MB/s, bzip2 around 15 MB/s, and LZMA around 20 MB/s. Your timings may vary by 1-2 MB/s depending on the data and depending on your CPU, but more or less, it's about that.

 

My not-very-special 7200RPM seagate disk which I use for backups delivers 100 MB/s (+/- 2 MB/s) for non-small-non-random reads (about ½ that for many small files). Both my OCZ and Samsung SSDs deliver 280 (+/- 0) MB/s no matter what you want to read, they'd probably perform even a bit better if I had them plugged into SATA-600 rather than SATA-300 (which the disks support just fine, only the motherboard doesn't). The unknown SSD in my Windows8 tablet delivers 120-140 MB/s on sequential, and 8-10 MB/s on small files.

My elderly close-to-fail (according to SMART) Samsung 7200RPM disk in my old computer is much worse than the Seagate, but it still delivers 85-90 MB/s on large reads ("large" means requesting anything upwards of 200-300 kB, as opposed to reading 4 kB at a time).

 

Which means that none of ZLib, bzip2, or LZMA are able to gain anything on any disk that I have (and likely on any disk that any user will have). Most likely, they're indeed serious anti-optimizations which in addition to being slower overall also consume CPU.

 

Speed-optimized compressors such as LZF, FastLZ, or LZ4 (which also has a slow, high-quality compressor module) can decompress at upwards of 1.5 GB/s. Note that we're talking giga, not mega.

 

Of course a faster compressor/decompressor will generally compress somewhat worse than the optimum (though not so much, really). LZ4HC is about 10-15% worse in compressed size, compared to ZLib (on large data). However, instead of running slower than the disk, it outruns the disk by a factor of 15, which is very significant.

If decompression is 15 times faster than reading from disk, then you are gaining as soon as compression saves you 1/15 (6.6%).

 

Also, given that you copmress individual resources or small sub-blocks of the archive (again, for speed, otherwise you would have to sequentially decompress the whole archive from the beginning), your compression rates are very sub-optimal anyway, and you may find that the "best" compressors do not compress much better than the others anyway.

For example, a compressor that does some complicated near-perfect modelling on 5-10 MB of context is simply worth nothing if you compress chunks of 200-400 kB. It doesn't perform significantly better than a compressor that only looks at a 64kB context (a little maybe, but not that much!).

 

LZ4 compresses pretty poorly in comparison with better compressors (but then again, given the speed at which this happens, it's actually not that bad) but LZ4/HC is very competitive in compression size. Of course, compression is slow, but who cares. You do that offline, once, and never again. Decompression, which is what matters, is the same speed as "normal LZ4" because it is in fact "normal LZ4".

Edited by samoth
1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0