Archived

This topic is now archived and is closed to further replies.

Virtual filesystems...

This topic is 5626 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I''m not completely sure I know what I''m talking about, but anyways... When I say virtual filesystem, i mean like .mpq files or .pak files, where one file has many files withen it, and the game uses the files inside. I just got my custom file-within-a-file program to work! It loads and unloads files perfectly! But I don''t want to have to extract each file at runtime. How do other games handle this? Does the game extract the files it needs and saves them as temporary files? Does it have special loading code to deal with the file-in-a-file part? Or is there a way to treat a file within a file as a normal file?

Share this post


Link to post
Share on other sites
well, i''ve never done this before, but i don''t see why you can''t open the PAK file, move to the correct position for the beginning of the "file" you need, and then treat it the same way as usual... for example, say you have a bitmap that starts at file position 1000, and is 500 bytes long, just open the file, move to the 1000th byte, and then use the next 500 bytes of that in your bitmap loading functions.
do i make sense?

Share this post


Link to post
Share on other sites
Well, most of the games have a wrapper library for file access functions. These wrapper functions internally uses different techniques to actually access the files. The main game/app code need not worry about how/where these files are accessed/stored.

Games using PAK files (or similar ones) generally have their unpacking and reading code inside these wrapper file access functions. On PC these wrapper functions may depend on fopen/fread etc. (or similar) commands for actually reading a file from the disk, but they may not directly return the data read by the fXXXX functions.

You can have these wrapper functions to load the entire PAK (not a good idea though!!) into memory and uncompress it (if it is compressed). Then any subsequent file access functions can return data from the memory.

Usually games do not employ ''writing-to-pak-file'' technique. All the files opened for writing should ideally be stored as seperate files on the disk instead of packing it into a PAK file. This is done for performance reasons and most games have only the game statistics to be written to a file, so it''s better off written to a seperate file rather than packing it into a PAK file. The performance benefit from this method is more than the PAK file advantage.

Games generally use PAK files to reduce the actual number of file system access calls (anybody thinks other way?) and to provide a better porting flexibilty to other platforms.

hope this helps...

Share this post


Link to post
Share on other sites
i implemented a simple vfs class that doesn''t support compression or encryption but nonetheless allows to dynamically write to pak files, so i''ll throw in some ideas.

wrapper library: absolutely. reading stuff from pak files should be no different than reading it from c streams or iostreams.

performance: this is a tricky question. for one, you want to write code as close to the operating system as possible to avoid various overheads. if you think of it, all file i/o libraries come with their own buffering code that takes memory and time to execute. for a typical program using fstreams, for example:

win32 caches the file in memory.
win32 performs buffered reads from disk in ReadFile. this is the second copy of the file in memory. time is spent for buffering.
c runtime performs buffered reads via fread. another set of buffers is allocated, more code is executed.
fstream has its own set of buffers on top of crt ones. this is the fourth copy of requested data in memory.
fstream reads the data into a buffer provided by you. your code allocates memory for the fifth copy of the data you''re reading.

the first optimization step would be to completely get rid of all c and c++ i/o routines and work directly with win32 ReadFile. even better idea, however, would be to use file mapping objects, thus avoiding buffers at ReadFile level. lookup CreateFileMapping on msdn. my vfs code provided pointers to (hopefully) the only copy of the data in memory to the application. you get both memory savings and performance boost.

an issue that didn''t occur to me until recently was the fact that it may be faster to store all data in compressed form and decompress it at runtime than load uncompressed data, if the medium is slow (ie, a cd drive). to accomodate compression (or any operations on the data, for that matter), my code would probably need close to a complete rewrite.

it''s important to note that on nt platforms opening files takes a while due to rights validation and other security code being executed, so loading a bunch of small files from a single pak file is likely to be faster than accessing filesystem for them.

loading entire pak file in memory: good if the files are small, bad if the files are big. unacceptable for files weighing at a hunderd megabytes or more.

writing to pak files: since my pak file structure was very simple and the library did not support any operations on data, implementing writing was pretty easy. if you, however, want to optimize access patterns or allow arbitrary data filters, it''s best to make a dedicated compression program and let the game only decompress files.

cross-platform code: i''m convinced that to achieve highest performance of vfs code you need to get as platform-specific as possible. writing portable wrappers is a good thing, but i will never write a vfs class in ansi c.

there''s a three-part vfs article on flipcode with source code, supporting artibrary filters for compression/decompression (i think), and some more stuff on google.

---
Come to #directxdev IRC channel on AfterNET

Share this post


Link to post
Share on other sites