• Advertisement

[.net] SharpFS (alpha release)

This topic is 4273 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I've been working on a virtual filesystem library written in pure C#, rather than writing a wrapper around some other unmanaged library (usually PhysicsFS). That work has finally come to a state where it's usable and functional, so I figured I'd toss an alpha out here (licensed under LGPL for now). I warn you right now this is very "fresh" code, and there are probably more than enough bugs, particularly given the sheer number of weird edge and corner cases that can be thrown into code like this. So apart from the fact that it's pure managed code, there are a number of neat advantages to this library. It's designed to be extremely flexible in terms of allowing various mount patterns. It forces you to mount something to the root of the filesystem (which is UNIX style, a single forward slash), but that root can be any supported archive type. Multiple physical archives can be mounted at the same virtual location, so if you have pak0.zip and pak1.zip, both can be placed at /data. You can mount something at /A/B/C even if A and B do not have anything mounted on them. You may subsequently mount something at /A/B without damaging anything, and in case of conflicts only the deepest virtual mount will be checked (in this case, /A/B/C will shadow any C that is generated from the mount at /A/B). The archive support is plugin based. To date I have only written two archive plugins, one for directories (ie standard physical filesystem) and one for zip archives. I'm using #ziplib for zip support. It also has support for tar, gz, and bz2, but I haven't gotten around to writing those plugins yet. You can also write your own plugins without making any modifications to the SharpFS sources (which lets you dodge licensing issues). Note that by default, the code does not actually load any archivers. You provide the ones you want in the constructor, or you can register them later. Oh, and a side note here: Make sure you include the correct archiver for the root in that constructor, or it will fail. (This allows optimal performance, since it's unlikely that you need to handle every possible archive type for any given project.) There's another class included, which is a layer built on top of the virtual filesystem. It provides a file manager that can queue up and background load a whole bunch of files at once. Both the virtual filesystem and the file manager are completely thread-safe for all operations. The file manager also prevents files from getting loaded multiple times (but you can release files if you like). Now, some of the downsides. Again, this is seriously undertested code. Everything is supposed to be thread-safe and exception-safe, but those things are easier said than done. There are probably mistakes in there which I haven't caught yet. I also am not sure if there are corner cases involved with mounting that I haven't dealt with. There isn't exactly a huge range of archive types supported, and really the base interface for archive plugins isn't quite stable yet so writing a new plugin should probably wait. Current directory (".") and up one directory ("..") are rejected outright at the moment, and I need to add support to handle them safely. Despite these problems, I'd really appreciate it if you guys would try this thing out. The more people who use it, the quicker the major bugs will be found and the sooner this will go from an alpha prototype to a fairly stable library that people can use. You do have a direct line to the main developer, after all, and so bugs can go from found to fixed rather quickly. Well I think I've rambled quite long enough. There's an online browsable source tree, or a downloadable tarball. Both are tagged to a subversion repository, so you can always hit these links for the latest revision. [Edited by - Promit on May 7, 2006 2:36:26 AM]

Share this post

Share on other sites
Advertisement
Usage sample:
//set up file managementFileSystem = new SharpFS.VirtualFS( ".", false, typeof( SharpFS.DirectoryArchiver ), typeof( SharpFS.ZipArchiver ) );FileManager = new SharpFS.FileManager( FileSystem );FileSystem.Mount( "/Textures", "Textures.zip" );//precache some thingsFileManager.Request( "/tree.fx" );FileManager.Request( "/Textures/tree.dds" );FileManager.Request( "/Models/tree.e61" );//get the actual data for the tree textureSystem.IO.MemoryStream stream = FileManager.GetStream( "/Textures/tree.dds" );Texture treeTex = new Texture( device, stream );//if you don't feel like using the manager, direct access is fine tooSharpFS.VirtualFile file = (SharpFS.VirtualFile) FileSystem.OpenFile( "/Textures/tree.dds" );//there's current directory support too (Note: The FileManager does not handle this correctly)FileSystem.CurrentDirectory = "/Textures";SharpFS.VirtualFile file2 = (SharpFS.VirtualFile) FileSystem.OpenFile( "tree.dds" );

[Edited by - Promit on May 7, 2006 2:27:23 PM]

Share this post

Share on other sites
This is great, I was just looking for something like this to be used on one of my projects. Great timing.

One thing I'd like to suggest howerver is the addition of aliases (or shortcuts). For example if you have directory /A/B/C and I simply want to refer to it as abc, then something like:

FileManager.AddAlias( "/A/B/C", "abc" );// then doFileManager.Request( "abc:file.txt" );// instead ofFileManager.Request( "/A/B/C/file.txt" );

I would add it in myself, but I figure if it's a feature you want you can put it in "officially" or something.

Share this post

Share on other sites
I made a semi-major change now, which is that the entire virtual filesystem deals with objects instead of Streams now. This cleans up a few things and more importantly introduces some interesting opportunities for clever things I'm exploring. More to come.

Share this post

Share on other sites
Quote:
 Original post by IFooBarThis is great, I was just looking for something like this to be used on one of my projects. Great timing.One thing I'd like to suggest howerver is the addition of aliases (or shortcuts). For example if you have directory /A/B/C and I simply want to refer to it as abc, then something like:FileManager.AddAlias( "/A/B/C", "abc" );// then doFileManager.Request( "abc:file.txt" );// instead ofFileManager.Request( "/A/B/C/file.txt" );I would add it in myself, but I figure if it's a feature you want you can put it in "officially" or something.

I think it would be cleaner as FileManager.Request( "abc/file.txt" ); that way it would treat the alias as much like a 'real' directory as as possible.

Share this post

Share on other sites
Quote:
 Original post by nagromoI think it would be cleaner as FileManager.Request( "abc/file.txt" ); that way it would treat the alias as much like a 'real' directory as as possible.
Same here, and that's how I'm going to implement it. I just need to work out a few details of exactly how is best to add the support.

Share this post

Share on other sites
I just thought I'd plug Tao.PhysFs (.NET bindings to PhysFS). Its source is available here and will be bundled with the next Tao release.

Share this post

Share on other sites
Quote:
 Original post by Rob LoachI just thought I'd plug <.... and will be bundled with the next...

Rob, while I do appreciate all the Tao work you have done over the years, that was a pretty pants thing to post in this thread.

As to the actual OP. This looks excellent at first glance. Raw maybe, but very promising and nice design so far.

This is something I will be looking to implement within our upcomming project.

Share this post

Share on other sites
Quote:
 Original post by JoviexRob, while I do appreciate all the Tao work you have done over the years, that was a pretty pants thing to post in this thread.
I agree, but it's just that I didn't have the time to give a proper look at SharpFS at the moment. I'll give it a good look at once I have the time.

Share this post

Share on other sites
Ok, aliases work now. Use the VirtualFS.MountAlias function to create them. (And it doesn't check for loops, so for god's sake be careful.) Syntactically they are identical to everything else.

I've done a ton of internal structural work now that makes this thing far more flexible. I'm still exploring exactly how much is possible, but I know for certain that all of the following are doable:
* Mounting HTTP and FTP directories (this is now implemented, see WebArchiver)
* Mounting databases (includes the ability to issue SQL queries to the database!!)
* Object filesystems (a la The Nebula Device)
* Interpreter based object database (see 1 2 3 by Washu)

Washu: All glory to the Washu!

[Edited by - Promit on May 8, 2006 8:33:48 PM]

Share this post

Share on other sites
Will someone humor me and tell me why we need a virtual file system instead of using the perfectly good one provided by the OS? I'm sure there is a good answer since I've seen several of these things about I just have never seen them spelled out.

Share this post

Share on other sites
Being able to read directly from compressed files for one (ie. mount a .zip file). It can also make including custom content easy, want to include some custom textures? Mount the directory containing them into the location you would normally read textures from, nothing else changes. You could have a 'current_level' mount where you can read all the data for the current level, wherever it may be, without having to pass paths around. The 'saves' mount could point to the current user's 'Application Data' directory. I can't think of any situations where a virtual filesystem is necessary, but they're very useful when it comes to managing your data.

Share this post

Share on other sites
Quote:
 Original post by thezbufferWill someone humor me and tell me why we need a virtual file system instead of using the perfectly good one provided by the OS? I'm sure there is a good answer since I've seen several of these things about I just have never seen them spelled out.
Probably the most noticable modified file system in a professional game was that found in the Quake games. The system read in "pak" files which were actually just zip files. Whenever you wanted to install a mod, you'd just drop the mod's pak file in the directory and away you went. If this pak file system wasn't implemented, you'd have to extract the mod's content into Quake's data folder, potentially changing and loosing the game's original content.

Taking that into consideration, a file system like this is really good for organizing game content. For example, you can have level content files seperated by zip files, but still maintain a simple interface in game code. Here is what it would look like in the file system if your level data was seperated by zip files:

BaseData.zip   - gaphics/player.png   - audio/playergrunt.oggLevel1.zip   - graphics/dog.png   - graphics/plant.png   - graphics/walltile.png   - audio/music.mod   - audio/dogbark.ogg   - scripts/level1.scriptLevel2.zip   - graphics/boss.png   - graphics/water.png   - audio/intensemusic.mod   - audio/bossrawr.ogg   - scripts/level2.script

As you can see, all content is organized by level, making it really easy to add new level content. Although this is good for adding new levels, what's import is how it appears to you in the virtual file system in code...
Graphics/    player.png    dog.png    plant.png    walltile.png    boss.png    water.pngAudio/    playergrunt.ogg    music.mod    dogbark.ogg    intensemusic.mod    bossrawr.oggScripts/    level1.script    level2.script
When you look at it in how the virtual file system would interpret it, it seems to combine all of the zip files into one managable interface. Since you assembled the zip files by content type, the virtual file system reads it as such, making it very easy for you to read through the virtual Scripts directory for all scripted content.

Share this post

Share on other sites
not to mention, checksum checking a single zip/pak file is a hell of a lot easier to verify proper installs rather than 10k content files.

Share this post

Share on other sites

• Advertisement