Sign in to follow this  
Basiror

Proper virtual filesystem implementation

Recommended Posts

Hi I am beginning to create a virtual filesystem, but there a few points I am undecided about. So how would you engage these issues: 1. Loading archives(zip files): a) load the whole archive into memory until you are done reading from it? b) only load the archive entries into memory one demand 2. Runtime additions One thing that is often used in modern engines is the direct file transfer from server to client. This however requires adding new archive files to you file hierarchy. I build my file hierarchy once at startup a) find each archive file in mod folder b) add each file to a list c) find each archive in the main game folder and add files to the list as you see I don t allow the use of none archived files for game content only config files and such can be read directly from the HDD so if id receive a new archive from the server I would add the files contained in this archive to the top of the file list however with this design I might get trouble with file versioning how would I know which files I may use to load a scenery after restarting the game? comparing the timestamps included in each zipfile entry could be problematic especially when someone hacks them another solution I came up with would exchanging some information about each file between server and client at connection time Informations: CRC32 checksum adler32 checksum filename timestamp This however requires the use of 2 seperate filesystem one for game content which is initialized at connetion time and one for gui content that cannot be used for cheating which is initialized at startup time or I build in a mechanism to identify the critical files with the information provided by the server thus I don t have to build the filehierarchy twice Any suggestions Thx in advance

Share this post


Link to post
Share on other sites
I have been working on a VFS myself. It doesn't allow "zip" files, but i do have my own archive files in there.
The idea being that you mount all your filesystems ( archives and regular folders, mabe even a network filesystem to the server).
A mount can take place whenever, so you can mount your patch, findout the sever has a new version, unmount the patch, update the patch and remount.

How do you deal with file versions? simple, make a scheme to mount the newest files last, and if multiple files exist in the VFS at the same location
(ie data\textures\wood.tga) then the one mounted last gets used.

What about loading the archives? I think the best option is to load all the archive's file hiarchy and keep it in memory,
but load the data in the archive only when you are going to use it.

My VFS is hosted here (vfs.zip)

There was a post a while back linking to persil's VFS, which is another poster's take on the VFS.
This one actually implements a zip file and custom archives. (lots more than my toy system)

There is also the old flipcode article wich could be of some help too.
It comes with sourcecode that is not OOP, and thus fairly static in places (no room for expantion),
but it does have some good ideas to get you started.

Share this post


Link to post
Share on other sites
Quote:
Original post by Basiror
Hi I am beginning to create a virtual filesystem, but there a few points I am undecided about.

So how would you engage these issues:

1. Loading archives(zip files):
a) load the whole archive into memory until you are done reading from it?
b) only load the archive entries into memory one demand


It would make things easy if you can specify a file like this:
file://localhost/c/Games/MyGame/data.zip/level1/lvl1.map
http://game.server.com:4711/data/level2/lvl2.map
file://localhost/c/Games/MyGame/mods/mod1/mod1.zip/data/main.tar.gz/readme.txt

When opening a file, create a filter chain (i.e. for data.zip you would insert a zip reader). Then every file provides the same operations, independent on how and where it is stored. Don't load the whole file into the archive. But you can provide some caching for compressed files. This will speed up access to already uncompressed data. (The cache would be another type of filter.)


Quote:

2. Runtime additions
One thing that is often used in modern engines is the direct file transfer from server to client. This however requires adding new archive files to you file hierarchy.

I build my file hierarchy once at startup
a) find each archive file in mod folder
b) add each file to a list
c) find each archive in the main game folder and add files to the list

as you see I don t allow the use of none archived files for game content
only config files and such can be read directly from the HDD


Why not let each mod provide a little config file, so that you can load and run the mod. Then the mod can load all of its files like shown above (maybe also by relative paths).

Quote:

so if id receive a new archive from the server I would add the files contained in this archive to the top of the file list

however with this design I might get trouble with file versioning
how would I know which files I may use to load a scenery after restarting the game?

comparing the timestamps included in each zipfile entry could be problematic especially when someone hacks them

another solution I came up with would
exchanging some information about each file between server and client at connection time

Informations:
CRC32 checksum
adler32 checksum
filename
timestamp

This however requires the use of 2 seperate filesystem

one for game content which is initialized at connetion time
and one for gui content that cannot be used for cheating which is initialized at startup time

or I build in a mechanism to identify the critical files with the information provided by the server thus I don t have to build the filehierarchy twice


Any suggestions


Thx in advance


There is no way to prevent cheating. But you can make it harder to cheat. The only thing you can rely on is your server. Let the game do requests (i.e. for moving the player), the server will only handle requests that seem valid (i.e. don't allow the player to fly, unless it is superman).

Checking the files is nice but can be hacked. But checking downloaded files before executing them is mandatory. You can use MD5 or something similar. This will also (almost) prevent you from having files with the same hash. Or you can create a GUID for your files ;-)

Versioning your files creates some other questions. How will you define compatibility ?

Share this post


Link to post
Share on other sites
@KulSeran:
The problem with mounting latest files last is that you can loose the timestamp of files(for what ever reason: e.g. transfer via some data devices that do not provite timestamps)

Zipfiles provide you a timestamp with won t get lost
Also mount thing would result in code coupling of the filesystem code with network code, not an option I think


Did you ever mess around with memory mapped files? to speed up loading times?
One could apply this to uncompressed archives

My current plan would check wether you are in developermode or not
and in normal mode it only loads .zip files
config files and such are read independent from the VFS, or maybe I implement a list of non VFS files

@nmi:
I already support mods.
you can specify the main game and the mod folder at startup with a config
but the vfs loads the mod folder files first

As for cheating prevention, I think if I make it difficult enough to hack such a system the amount of users willing to hack it will rapidly decrease

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this