Sign in to follow this  
Shael

Flexible File System

Recommended Posts

I'm trying to figure out a good way to handle paths in my file system and thought I'd see what other people are doing and what is done in commercial games/engines.

The problem I'm facing is if I pass in a path to say a mesh ("meshes/rocks/bigrock.msh") the resource loader would look at the current directory plus that path and it would find the file. But then the mesh requires textures/materials and other resources within its own directory. These won't be found unless I programatically change the working directory to "meshes/rocks". This works but it doesn't seem very flexible and it kind it requires keeping track of what the current working directory is. I'd also like it to tie in nicely with a virtual file system (pak files).

Is there a better way to handle this sort of thing? Perhaps a list of "search" paths within the resource manager and a stack of temporary paths that can be pushed/popped when loading particular resources that require resources within their directory and external directories too?

All ideas welcome!

Thanks

Share this post


Link to post
Share on other sites
Sound like you just need to separate directory and file name. Instead of just a file name containing both a directory [i]and[/i] a file name, separate them into two separate variables. The path stores the base path where you want to look for your resource, and the file name pointing to the particular resource. If you need indirect resources, you have the base path already, and just tag the new resource file name onto the base path. The file name can even contain further directory structures if you want to organize sub-resources into subdirectories.

Share this post


Link to post
Share on other sites
[quote name='Shael' timestamp='1306504263' post='4816424']Is there a better way to handle this sort of thing? Perhaps a list of "search" paths within the resource manager and a stack of temporary paths that can be pushed/popped when loading particular resources that require resources within their directory and external directories too?[/quote]
What Brother Bob says it's a clean solution. Personally I went for a dirty solution.

[u]Considering that this is not a performance path[/u], there's a simple and effective way.
RAII an object getting the current dir path ([i]cwd[/i]), on destruction this object changes directory to [i]cwd[/i].
RAII another object changing to the directory you need to access to load [i]resourceA[/i] ([i]dirA[/i]) (those two can be the same object)
Load [i]resourceA [/i]using the appropriate calls.
Now, [i]resourceA [/i]needs [i]resourceB [/i]to load.
Just call the manager. It will internally RAII similar objects and load.
When [i]resourceB [/i]is loaded, it will return. RAII'd objects will pop off and the previous dir ([i]dirA[/i]) will be restored.
The previous manager will now keep loading [i]resourceA[/i]. It has no idea of anything that happened... and it will not care.
When done, it will return [i]resourceA[/i], its directory object will scope out and the previous dir will get restored to [i]cwd[/i] (in my case is the "generic content" path).

It's a bit dirty but it's easy and flexible. A good "fire and forget" solution for me.

This approach tends to fragment the heap a bit because of the heap allocations to hold the path strings but I haven't found it to be a problem for the time being and I don't expect this problem to arise any time soon. I suppose it's somehow possible to extend this to handle packages... somehow.

Share this post


Link to post
Share on other sites
@Brother Bob:

What you've suggested is similar to what I'm doing except I only pass one path to a resource and extract the filename out of it to get the local directory of the resource. I also store a "base directory" within the file system which is essentially the root directory of the engine. I think the problem is the need for search paths, kind of like the environment variable PATH in windows. That way it could search locally first, then search other paths specified. Perhaps each resource type can specify search paths and there could be some generic paths held by the resource manager.

@Krohm:

I think what you're referring to is the stack idea I mentioned in my first post? The pushing/popping of directories I guess is sort of what I'm doing currently by programatically changing the CWD to the resource directory, loading the resource, then pop it back to previous CWD. This doesn't quite work 100% because if I load a resource from "basedir/resource/resourceA" and that resource specifies a resources called "shiny.fx" which is located in "basedir/effects/shiny.fx", the resource manager wouldn't know where to look as it wouldnt be in the pushed local directory of "basedir/resource" and it wouldn't be located in the popped path of "basedir/shiny.fx". It'd only work if resourceA specified the path to be relative to the basedir eg. "effects/shiny.fx".


Another I idea I had was to simply make every resource always specify its child resources relative to the base directory and then whenever trying to load a resource it'd simply be basedir+resourcepath. This seems kind of limiting though and prone to errors.

Share this post


Link to post
Share on other sites
[quote name='Shael' timestamp='1306550042' post='4816648']I think what you're referring to is the stack idea I mentioned in my first post?[/quote]Yes, pretty much.

[quote name='Shael' timestamp='1306550042' post='4816648']The pushing/popping of directories I guess is sort of what I'm doing currently by programatically changing the CWD to the resource directory, loading the resource, then pop it back to previous CWD. This doesn't quite work 100% because if I load a resource from "basedir/resource/resourceA" and that resource specifies a resources called "shiny.fx" which is located in "basedir/effects/shiny.fx", the resource manager wouldn't know where to look as it wouldnt be in the pushed local directory of "basedir/resource" and it wouldn't be located in the popped path of "basedir/shiny.fx". It'd only work if resourceA specified the path to be relative to the basedir eg. "effects/shiny.fx".[/quote]This happens because your assets are poorly conditioned - you have not established a directory convention for your system.
You naturally put all .fx files in a sub-effect directory. Now, what's the point of trying to load a resource which is know to stay in the [i]effects [/i]sub dir from the [i]SomeOtherKindOfResource [/i]directory? There's none.
Each manager should work in its own directory.

[quote name='Shael' timestamp='1306550042' post='4816648']Another I idea I had was to simply make every resource always specify its child resources relative to the base directory and then whenever trying to load a resource it'd simply be basedir+resourcepath. This seems kind of limiting though and prone to errors.[/quote]Here is the "conditioning" above. What limitations do you see? What problems do you see?

Share this post


Link to post
Share on other sites
My favorite system is to simply not expose the idea of 'paths' on the game side -- only filenames. Directories are only used as a convenience for the developer to keep things organised and are ignored by the game.
The downside is that you can't have two files with the same name in different directories (one will be ignored), but I've not found that to be a problem in practice. At the same time, this is an upside in that a model can store a link to "[font="'Courier New"]concrete.texture[/font]", and if you decide to move which folder that texture is stored in, the model is still valid.

e.g. given a data directory containing:
[font="Courier New"]data/levels/01/building01.model[/font]
[font="Courier New"]data/levels/01/concrete.texture[/font]
[font="Courier New"]data/shared/bricks.texture[/font]

The game will only see:
[font="Courier New"]building01.model[/font]
[font="Courier New"]concrete.texture[/font]
[font="Courier New"]bricks.texture[/font]

Share this post


Link to post
Share on other sites
You could take a look at both Ogre and PhysicsFS.
Mount points / aliases are also an option together with "real" paths (ex. "Textures:Face.jpg") with multiple real paths for each alias, making moving of a subfolder as easy as adding another path.

Share this post


Link to post
Share on other sites
I like the idea Hodgeman has proposed and looks to be akin to what PhysicsFS does.

The only problem with this is it wouldn't work well for non-game things like asset tools built upon the engine. Eg. world editor/model viewer. You wouldn't be able to just drag and drop or browse to an asset located elsewhere on your machine because you are restricted to the game/engine directory. Could maybe add the ability to override the loading function to be able to specify an absolute path to get around this use case.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this