Loading file paths and how to access them

Started by
25 comments, last by Kylotan 6 years, 11 months ago

Hello forum!

Quick disclaimer, I think this topic is language agnostic but whenever I refer to programming concepts (classes, objects, instantiating etc.), they will refer to C++.

How are you handling fixed file paths for your software/game? E.g., you want to load a texture, how do you tell your load-system where to find all textures?

I do not want to hardcode my paths, so I would be happy to hear some more flexible approaches. Especially moving them to a file that contains all of them. The actual issue is more on how to pass them around.

Do you instantiate one class at the very beginning and simply access your "file-path"-file? Do you pass this around? Did you consider working with globals for this? Or do you simply instantiate the object again and again wherever it is needed?

I'm not really sure if I want them to be part of the dependency injection, but that is after all just a weird feeling. It might be the only sane choice in the end.

Thanks for your time!

Advertisement

I create:

  • A BlobLoader, which is abstract (could be a directory, a compressed package of assets, a network file system, etc).
  • Many different factories for loading different types of asset. The blob loader is passed into the constructor of these.
  • Several "asset scopes" depending on need. These control the lifetime of loaded assets (when the asset scope is deleted, all assets that were loaded into it are deleted), and also act as a cache to prevent duplicate loading. Each asset scope can be passed another one upon creation to use as a 'parent', which is checked first during cache lookups..

I don't refer to files by a path, but just by a generic name, which is the hashed filename. e.g. MyTexture.png might get hashed to 0x12345678. The different implementations of the BlobLoader interface can figure out how to retrieve the actual asset data from these asset names.

To load a file you then need a pointer to the appropriate factory. I make an ugly god object for many of the widely used game systems, and pass this around as an argument to Update/etc, so it's available to the main entry point of the gameplay code. Other systems will create factories internally, which means they will need the BlobLoader passed into their constructors... which isn't a big deal. It's waaaaayyyy better to pass these dependencies into constructors than it is to create hidden dependencies throughout your codebase via globals.

e.g.


//god object :(
GameSystems::GameSystems(...blah...)
  : m_blobs( new BlobLoader(...) )
  , m_window( new OsWindow(...) )
  , m_gpuDevice( new GpuDevice( *m_window, ... )) )
  , m_textureFactory( new TextureFactory(*m_blobs, *m_gpuDevice) )
  , m_materialFactory( new  MaterialPackFactory(*m_blobs, *m_gpuDevice, *m_textureFactory) )
{}

GameLevel::LoadLevel(GameSystems& sys)
{
  m_myMaterial = sys.m_materialFactory.Load( AssetName("myMaterial.mat"), m_levelScope );
}

Oh, interesting.

But I do not really understand what a BlobLoader is after all.

My exact problem is that during prototyping, I called my file-loaders like this:


load("asset/texture/" + texture_name)

But I now I would like to change this to:


load(textures + texture_name)

I usually pass dependencies via constructor already : )

So, the BlobLoader implements the abstract behaviour of loading to the factory, is that right to assume?

m_myMaterial = sys.m_materialFactory.Load( AssetName("myMaterial.mat"), m_levelScope );

Let me attempt to assume the further loading-procedure, if the object is not cached. There is probably a moment, where the factory will call "load" of its LoadBlob, and the LoadBlob pretty much implemented a custom behaviour.

If the LoadBlob is nothing but a packed set of data, it will search through it after content whose name is identical to the given hash. It might also be only another loading-call to the OS, providing an already implemented file-path-prefix.

This sounds really neat, would have to restructure my code though, I have a namespaced file with commonly used file-system calls to the OS, e.g. Load-File-Content, which felt pretty KISS to me. But it requires a full file-path from the initially calling class.

There is actually quite a bit written about that sort of loader that hodgman described if you want to look it up. Right now I only load textures via stb_image.h and text files. I am in the future going to do as hodgman describes so it's easier to package my assets up for distribution and for on multiple systems (mobile, console, PC/Mac, etc) without having to do much more than mentioning the filename and by extension it handles it magically for me.

Different OS's have different rules for file locations though. On PC I keep mine under the default directory (GetDirectory()) under assets\textures\some_texture.png or assets\map\some_map.txt, etc. But I know you're suppose to save, save files to a certain directory to comply with Windows rules (though nothing is currently stopping you from doing differently).

Two options I can think of for you off the top of my head:

1) Pass in a list of file locations to look in, and run through it until you find the asset or fail. So you would get a directory "assets\fonts\" and append the filename to that.

2) Pass in the list like before but tell it what each location handles extension wise to limit searches.

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety." --Benjamin Franklin

Hard-coded asset paths aren't necessarily a problem if they are considered paths within an abstract asset database rather than a filesystem. The default implementation may well be a filesystem, but as a trivial improvement it's very common to have a virtual filesystem that automatically treats zip files (or other archives) as directories, or one where you explicitly mount zip files (to avoid ambiguity of the automatic approach). It's also common to have multiple targets available for a given path with different priorities (e.g. so that a new asset can be tested locally before committing it into the built data archive). This can be useful for more than art assets (e.g. configuration files).

There is actually quite a bit written about that sort of loader that hodgman described if you want to look it up

I tried finding it, what is the actual name of this loading-concept? Blob Loader gave me no results.

Thanks everyone! This gave me a new point of view onto the loading-process, I will dig deeper into this, too. :)

But I do not really understand what a BlobLoader is after all.
So, the BlobLoader implements the abstract behaviour of loading to the factory, is that right to assume? Let me attempt to assume the further loading-procedure

Sounds like you understand :)
Yeah my Blob Loader is an abstract interface that you can give a name to, and it gives back an array of bytes. When paired with a factory, you can convert a name into some kind of useful object instead of just bytes.

For my simplest blob loader, you pass a base path into the constructor and it reads files in that directory. Other implementations could read files out of a ZIP archive, or read files over TCP/IP (that last one is very useful when developing a game for another platform - console/mobile/etc)

Ah! So the byte-array is probably to generalise a type that all Blob-Loader can translate their "loaded" objects to. Using so many high-level-libraries makes it feel a bit weird to pass around byte-arrays. I'm really anxious about reinterpret_casts, haha :')

Just out of curiosity, are there any other ways to pass these loaded objects back to the factory? I mean, if I could make sure that every content/object loaded by every Blob-Loader is deriving from another base-class, I guess that could work. Nonetheless, I really don't want to and possibly cannot rely on being permitted to edit my libraries.

I totally agree my previous writers in what they wrote but have some kind of completition. For the purpose of going to multiple platforms I integrated a set of static functions that are inside the storage namespace handling some common type functions like creating/searching/moving/deleting files/folders and resolve special path requests for Application Path, Root Path and User Data where each path is equalized to use forward slashes everywhere (Windows doesnt has a problem with that) so query the application path may result in


C:/Program Files/myTest //Windows
/data/data/myTest.myCompany.com //Android

In the next step I set root directory (working directory in Windows) to something where my assets are and could then call


Storage::File::Open("./MyAsset", Storage::File::Open)

Finally anything in my environment uses streams instead of plain byte arrays where MemoryStream wrapps arround a byte array so they may be used too. Any stream is based on a general interface IDataReader/IDataWriter that supports Reading/Writing of a single byte, array of bytes and Position/Length properties where one specialization is StreamReader that wrapps arround another IDataReader for substream reading like in a package.


InFileStream ifs("./MyAsset", Storage::File::Open);
if(ifs)
{
   //DoStuff
   while(!ifs.Eof())
   ;
}
if(ifs)
  ifs.Close();

Is the standard way to open-read a file from disk in my case.

For production code it is totally ok to use direct file paths that come from an entry point file that you need. Otherwise your program wouldnt know what file to load and in the end where to start from. This entry point may be an asset package or an .ini file. That totally depends on your game environment.

This is a common technic used by for example The Elder Scrolls games to load the games content (Oblivion.bsa asset package as example) that may be replaced my modders to wipe change the game completely.

When you have got processed your asset entry point you need to give that to the asset manager for loading levels, main screen, whatever. I personally prefer asset hashes too as Hodgman already wrote because they have some advantages for protecting your assets (modders wouldnt now so fast what kind of file it is and what it is used for) and saving memory to use 4 bytes instead of a large indefined byte length string.

Again in production you could use strings or a file table hash <=> path conversion to get your assets loaded.

At least I would recommend to use some kind of internal managed asset storage (e.g. a package). I use an aligned, signed and encrypted custom format on this action but you may also use .zip for starting purposes. My format is build from 64 kb aligned chunks with an algorithm to minimize fragmentation to speedup asset reads from disk so the package handler contains a small filetable in its head that points into the chunk and offset. A stream finally needs to jump to position (chunk * 64kb + offset) and read N bytes equal to the file size.

Finally my asset reader functions are also static functions that take an instance of the abstract IDataReader (or sometimes also IDataWriter) interface putting out processed data like a texture from image or a mesh that may be handled however depending on the engine

Ah, that is similar to what I'm doing at the moment. I have no interfaces up yet, but consider doing so.

The issue I have is that the e.g. loading is often implemented by other libraries. Therefore, they will return a type that pretty much is ready-to-use. It feels weird to transform such to an e.g. byte-array and then cast it back to its original type. It feels like a weird overhead. I will need to do some benchmarks on how expensive this can become.

But I'm really thankful for your ideas and concepts that definitely changed my point of view on this topic overall.

This topic is closed to new replies.

Advertisement