For some time, I've been trying to figure out how to allow my game's data and logic to safely allow mod plugins similar to how Bethesda's work. Their method seems pretty sane, however I can't exactly see how they did it, nor do I have any connections to ask these sort of questions.

The issue is that say that the player already has a save game. And there exist an item in the save game such that... it has it's own instance data to work with that is completely unique to that one item. He installs a mod that adds more data onto the pre-existing mod. And then uninstalls the mod, and adds in a few more that also adds more data ontop of it.

What would be a sane and fast approach to this?

I've thought about using two different methods....

The first one which uses a monolithic class inheritance structure for low level system information that needs to exist. And then anything unique will be composited in with some identifiers. This method looks pretty identical to what bethesda's file format looks like.

So for example... assuming that [Engine Data] is required to exist.
ARMOR:

Harrowing Soul's Chest Plate: [Engine Data][Durability][Price][Description][Script]

Rogue's Hood: [Engine Data][Durability][Description][Hidden Description][Quest Reference]

Short Sword:[Engine Data][Durability][Price]

Where after the monolithic class, all extra data exist directly adjacent to the main class. I'd like to maintain data locality for the whole system so it's easier to serialize, pointer patch, and fit into tighter memory boundaries... but I'm getting closer to the point where I'm willing to deal.

This would also mean that scripting languages like Lua would need some special API to grab information that's not part of the main class already. But that won't be too much of a problem if I just cache the pointer.

The other thought was possibly using MongoDB, SQLite, or some other offline DB... but I'm not entirely sure about this. I've heard of games using Databases as save files and file formats. The prospect of being pretty easy to merge changes is also fairly nice. But it feels like peeling a grape with a sledge hammer to consider this. Not to mention I'm not entirely sure how fast using a DB would be in real time. I'm also not sure if they can load completely into ram either. I've literally never used a Database outside of web applications.

A side note, I'm avoiding the Entity-Component approach because the idea is to allow as much freedom to the users as possible without needing the source code. Even possibly exposing -most- of the engine's low level systems to Lua. Entity component requires things to be predefined, which is not the end goal here.

If anyone has suggestions, idea's, or better yet methods that works well for them, that would be helpful.

Use composition. It is super effective.

In this case you've got several different attributes: engine data, durability, price, description, hidden description, quest reference, and probably more.

Big relational databases work well for persistence, but realize that queries are time consuming. Even fast queries to an RDBMS take longer than most video game frames. If you're storing a lot of data and reading it as part of loading a character or map or level it can work, but they are a poor choice for this type of data.

In this case you might use an entity component system approach, composing your game objects with a bunch of different component pieces. That is a somewhat heavy solution as well. If it has a price component, do all the price component work. If it has a quest component do the quest component work. Components tend to be bigger pieces of data and tend to have more processing as big systems.

My first inclination is to use a plain old collection of attributes, a data dictionary of keys and values. If it has a price attached do the price related work. If it has a description attached do the description related work. If it has a quest reference attached do the quest related work. If it doesn't then skip it. When the object is created you can use the data values to create the instance and attach whatever pieces of data need to be attached.

Alright so I was origionally on the right track then. Well what I was thinking when it came to composting data was to do something along the lines of this.

struct Field_header {
    Field* Offset
    int Hashed_Name
}

Class Register {
    int Index_ID;
    int Number_of_Fields;
    Field_header Field_Registry;

    //Engine data goes here.
    
}
//Class used to keep item specific data.
Class Field {
    int Type_Flag;
    string Field_ID;
    char[] byte_buffer;
}

Honestly, The buffer bit is probably unnecessary. My thought was to allow the scripting language to speed it's self up a bit by allocating everything into a single buffer if it ever uses a lot of variables. But from the looks of things, Lua might not support such an action. I don't mind using Angel Script, but it's not clear on how userdata is handled. But the bright side is that it actually does support script introspection.

I would probably implement such a system by allowing entities to have a set of named properties. The property names would use namespaces and each mod would have its own namespace, so properties would not clash. Namespaces would probably be implemented as an integer ID so it wouldn't be too space-inefficient. If you are storing everything in this basic map/dictionary/lua-table-thing then the only problems you have are key clashes (solved with namespaces) and type checking (solved with a flag or whatever).

Also, immutable properties don't need cloning into every instance, and setting up a clear division between instances of X and the definition of X can save a lot of space.

The database issue is sort of unrelated, although I can see that you would completely sidestep the issue by having an arbitrary number of rows and a query language with which to examine them. However, storing things effectively on disk is not the same as storing them effectively in memory and while I think SQLite is potentially a great save-game back-end I think it's best used as a mirror of in-game structures, rather than being queried during play (at least when it comes to instance data like individual items).

while not directly related to data structures used, one thing i've noticed is it seems the lack of UUIDs can cause conflicts between different mods.

i suspect bethesda designed the engine to be moddable from the get go - to the point that the game itself is simply a very big mod. as are each of the expansions. its sort of data driven design taken to the extreme. with a generic highly data driven engine, modding is easy as the whole thing is driven by "mods".

Yeah... but I think that their custom scripting language makes it a world easier for them to support such a thing. After spending some time poking around their save structure, It looks like script data is stored per instance, and then loaded when called on.

Then again... this problem might very well be more simpler than I am giving it credit. The biggest issue is letting users add more data and remove data from the scripts without breaking the game engine. The above design was also with consideration that I'd like to try and thread the with OS threads game logic so it scales well. But after poking around, I think that might be unnecessary.

http://www.creationkit.com/index.php?title=Threading_Notes_(Papyrus)

Bethesda's threading method is starting to look more like coroutines than an actual thread. It's not explained very well, and it could very well be true threads.

[Only one thread can do anything at a time with an instance of a script. Whenever a thread first becomes active in a script, it "locks" that script, preventing other threads from accessing it.]

If it is co-routines, then I can probably get away with Lua or Python and allow them to store script data until it's time to save the game. It'd mean that I'd need to find a way to serialize data and more carefully reapply them on load time.

Big relational databases work well for persistence, but realize that queries are time consuming. Even fast queries to an RDBMS take longer than most video game frames. If you're storing a lot of data and reading it as part of loading a character or map or level it can work, but they are a poor choice for this type of data.

I don't know what database you're using, but on the servers I typically work with, the complete round-trip time of sending a simple, plain-text SQL query over the network to another server, processing it, and getting the result back often takes less than one millisecond. I'm sure a game could do it even faster with a local, in-memory database. And actually, I know that's true, because I've also worked on a game that did that.

Large databases with around 40TB of data presently.

You mention a millisecond.

Kept in local memory lookup, that is on the order of tens of nanoseconds.

Use the nanosecond-speed one when possible.

Large databases with around 40TB of data presently.

You mention a millisecond.

Kept in local memory lookup, that is on the order of tens of nanoseconds.

Use the nanosecond-speed one when possible.

So, you're saying that Tangletail's game's DB performance will be similar to a 40TB database, and that one frame of a typical video game happens in tens of nanoseconds?

No I did not wrote that or even infer that, but I can understand how one might jump to that conclusion.

RDBMS systems are slow relative to games, and that is by design. Even for tiny queries executed locally such operations take hundreds of microseconds, even milliseconds. Bigger queries take more time, milliseconds, seconds, and even longer. Their design is to process large bodies of data and return bodies of results in a timely manner relative for business purposes.

In contrast, the data that is typically accessed in games is best in main memory or caches, accessed in nanoseconds. There can be thousands or even millions of these tiny reads every second, and stalls for data reads are one of the most common bottlenecks in games that developers fail to account for. Keeping the CPU fed with a steady stream of data is a core task and responsibility in games.

The difference between the two is on the order of several hundred thousand, or millions or tens of millions of times slower. When keeping the CPU fed with fresh data is an important task in most games, a developer cannot choose the option that is four or six or eight orders of magnitude slower.

As I wrote, databases can work well for persistence and certain other data, but an RDBMS is a bad choice for a game's moment-to-moment data.