Last entry:

https://www.gamedev.net/blog/1930/entry-2260953-asl-geometry-tessellation-shader/

Despite being extremely busy the last weeks, I still found some time to put into developing the engine. This time around, I decided to tackle one of the ealierst features/problems I originally put in: The asset management.

The beginnings: K.I.S.S.

A short overlook of how the system originated. I recall that this was one of the very first things I built in. Which makes sense, you can virtually do nohing without having some sort of texture/mesh/... loaded. In keeping
with the KISS-principle, and the longing not to overengineer, I put a simple manual system, where for each type of resource, a few classes had to be declared:

- A cache class, which stores all loaded assets, and allows access by a key (which was usually a string name, since I mainly referenced assets manually back then). There is a templated class called core::Resources for this,
which acts like a std::map, with a few extra features. It is possible to declare a core::ResourceBlock, which records all loaded assets between an Begin()/End()-call (used for seperating asset handling for plugin/game/scene assets).
But the instance of this cache itself had to be instantiated manually, which usually happend in a Package class of the module (Gfx, Entity, Audio, ...), and be passed to the modules context as a member, so others can access it.

- A loader class. This had no regular interface alltogether, but usually it got a reference to the cache, as well as any other resources/assets it might need to access (material needs textures, etc..). It than has a load-method which
takes a file name parameter. Then, assets where ordered in a XML list file, like this:
ArkWalk ArkRun ArkJump ArkRunJump ArkShadow
This is due to the fact that first of all I didn't want to reference plain files directly as assets (since I've made very bad experiences with this in the Rpg-Maker), and also I usually had extra parameters I had to store, which
didn't fit in e.g. a plain png texture file. I had no toolchain back then, so I just manually put all assets in the list, which wasn't a big deal for the first few, small projects.
Again, this loader was manually instanced, placed in the Context-struct, and called when needed.

For a time, it was good...

... but after a while, I started to run into more and more troubles. Well, not exactly troubles, but as my editor/toolchain grew, and I ended up adding more and more types of assets, as well as working on a project which involves
thousands of seperate assets like textures, sounds, scripts... , it showed that the old system isn't sutaible anymore. So I decided to go a step forward, and adapat a dynamic, generalized asset handling like e.g. in Unity or Unreal4.
There are some very good, objective reasons for this:

- Less (duplicate) code produced. To be fair, the loader/cache-thingy isn't all that big of a deal. However, in editor mode, I had to declare so much things for every single asset, which couldn't even be generalized without having
all assets share some common code in the first place. I had to declare a seperate asset window, with a tree (which involves different callback-interface implementations), a pick-controller for selecting an asset in the rest of the editor,
etc. etc. . Having a general asset management would allow me to only generate the code for each asset manually, that is really needed (parsing the asset data, the editing/viewing tools, what Icons to use, ...). Creating new assets, deleting/renaming them, would all
be unified.

- Automatic import of new assets. Although I already managed to implement "Import X" functions for most assets, it still needed to happen in the editor, which specially made adding multiple assets tedious, as well as adding some to the
plugins (since I have no toolchain for those). With the new system, there will be no list in which assets are stored. Instead, on startup (in editor mode), the asset folder will be scanned for all files, which will then be available in the project.
I could have done that in the old system, but I would have had to write it for each asset seperately.

- Hot reload of assets. A small, but very important feature. I don't know how many times I've had to restart the engine just because I wanted to replace a texture with another. Again, could have been done in the old system, but with O(n) complexity for each asset.

- Partial saving of assets: Becoming more and more important as the project grows, in the old system I had to save each type of asset all together (since you can't just partially update an XML-file on the disk. Even worse, I had no "dirty"-flag mechanics,
meaning pretty much that the whole project had to be saved every time I pressed Ctrl+S. After a while, this does show, and I don't want to wait 5 seconds every time I save. With the new system, I can mark assets as dirty, which will then be saved seperately of everything
else.

- Binary packing: Another thing I didn't think about back then, was how to handle actually producing a distribution game package. I used to just give out the working project, with all the XMLs and the plain image textures. For both space consumption
as well as loading speed I want to offer packaging in an engine-specific binary format. Doing this with the old system was borderline impossible, but with the new asset system, data loading and asset generation is now seperated, allowing my to get my asset data from
whereever I want, without the loading routine caring (expect for whether or not its binary, ofc).

- Storing of user attributes. Actually, if I'm not mistaken I belive this is a feature that most other engines don't have, but I felt I really needed. When making my 2D Rpg, I always had the problem of "where do I store information about the character textures?".
For example, a character texture might have 1-4 directions (usually only 1 or 4), 1-X frames, and it will be displayed with a certain animation speed. Where do I store this information? Its not part of the base texture format, since I'm not making a
fully fledged 2d Rpg Engine. In the Rpg-Maker, this used to be part of the file name (f*ck me, if I ever changed the number of frames in a texture... not even talking about that this has no visual editor support, so f*ck it), so not an option. I first though
about having an external database, where you could link textures to this sort of information. But well, with the new asset format, its quite simple. I allow so called UserAttributes to be stored in the asset declaration. Those can be added by any plugin,
with an arbitrary name and type, which can then later be accessed by the plugin code, and of course edited in the editor. Due to natural loading order, only textures the are part of the project will have those attributes. A very good solution for the problem in my regards,
and spared me a few headaches.

- Reference counting/return of default values. A simple issue with my older system was that I didn't have any reference counting for my assets. Well, this usually wasn't a problem but it also didn't allow for any discarding of assets while e.g. a level is running,
so for that sake, introducing references via the generalized asset system for future unloading of unused assets to save memory is certainly a good thing. Also, I ran into trouble with loading order/missing assets, which my new system now allows me to completly surpass.
This is due to the asset system now being able to generate an "empty" asset with default values (checkerboard texture, empty script...) in case the asset is not being loaded while accessed. This means less hussle checking for nullptrs when cross-accessing assets,
as well as no more losing of stored references when somewhere an asset isn't loaded/isn't loaded in the right order.

- Removed the need for manual cross-access of assets and parsing of attribute types. Again, more of a code-duplication/complexity problem than anything else. So usually, assets would need to access other assets. In the old system, this would mean
I had to pass the cache-class to the loader, read out the key in the correct type, access it, usually check for nullptr-values & emit a warning if so, and so on. For attributes (texture size, mesh type), I had to access the text-value from eigther xml-node
or string, and parse it to the correct type, making range-checks for enums all the time, etc... . Now that I already have a global type system, both of those issues can easily be pulled out of the specific asset format. Every generic asset now has an "attribute"
header, where all those informations are. The name/type of each attribute is declared when registering the asset type, and the loading/parsing is done in the generic asset layer, so that the actual loader just gets whatever he needs via function parameters.
Way, way easier, and removed a lot of direct coupling between different asset caches/loaders.

Phew, I had much more to say about this than I originally planned to. But I also think its important to talk about this. I've heard many discussions concluding that generalizing asset management isn't worth it, and that you should just KISS. Well,
KISS is certainly the way to go for 1) starting a project and 2) for small/medium-scale (throwaway/one-time)-projects. For anything of larger scale, as you can see, there are many benefits for having a generic asset layer. As mentioned before,
you can certainly do all those features mentioned about if you have your seperate hand-written asset Cache/Loader-classes, but the amount of near-identical duplicated code you'd have to write is ungodly. I didn't fully count how much LoC I actually saved
from porting the system, but I think it was something along the lines of -10k (overall 180k for the engine right now), whereas the generic asset libary has around 3k LoC. Now I know this isn't the best of metrics, but you get the idea. Saving 7k LoC written
while providing a ton of new features is certainly a win.

The new asset file format:

Okay, I think this post is long enough already, but I just wanted to give a brief glimpse at how the new format looks like. For those of you already familiar with Unity/Unreal, it shouldn't surprise you that there is a file with the ending "aasset" generated
for every imported asset. This is how it looks like:
TextureAsset ArkHit ArkHit.png 0 0 None Unknown 0.0 3 4

For editor-mode assets, I decided to go with plain XML, for version-controls sake (we actually had real troubles due to Unreals binary assets in our recent university projects). So lets go over the format quick:

- The Type-Tag stores the type identification string, as registered in the type system. This will be used to lookup which type of asset has to be generated.
- The Name-Tag stores the name by which assets can be referenced. This is pretty much obsolete, I just had it there because it made porting a bit easier, since I've got to refine the whole access/reference part to support duplicate file names
in different plugins/folders, etc...
- The File-Tag references the data-file. As you will see in the implementation, the new asset format seperates asset and data generation. In this case, it references a texture file, but for anything that is already engine specific (prefabs, scripts...),
there is an alternative Data-Tag, which stores the asset data inline (so that I don't have to generate a seperate file for anything that will exclusively be edited by the engine). Having a File-Tag on the other hand is useful for images.
I also didn't like Unreals approach of storing all data completely inside the asset and not checking in the source file at all, since this way I often had to ask our artists to give me the texture seperately so I could make a small edit and reimport it.
- The Attributes-Tag stores the base attributes of the asset, which are declared at asset registration. Due to this declaration, the parser knows exactly which type it has to translate those values to. A complicated template-mindfuck will than expand
the parameter list, so that the loader gets those as parameters of the type he specified, but you'll see more of those in the future.
- The UserAttributes-Tag work similar to the normal Attributes, but as I mentioned its application/plugin-supplied.

And thats it. I currently don't have a binary format, but implementing this shouldn't be a problem.

So far, thanks for reading. Next time, I'll go into more detail about the actual implementation of the asset loading/handling.

template<typename Key, typename Asset>
class Resources
{
public:

// lots of utility-functions here

private:

    std::unordered_map<Key, sys::Pointer<Asset>> m_mAssets; // sys::Pointer is my unique_ptr implementation
}