Hello,

I'm making good progress with my game engines dll plugin system. Basically, nothing but one constructor function needed to be exported by a plugin to make it work, up until now. While seperating my modules into seperate dll projects, I've come across one issue again I've ignored earlier. Now its really turning out to be a big problem. Take a look at a sample code from the shadow plugin:

	        auto vEntities = m_pEntities->EntitiesWithComponents<ShadowCaster>();

		size_t count = 0;
	        for(auto pEntity : vEntities)
	        {
                                // do other stuff here ...

				if(auto pLight = pEntity->GetComponent<Light>())
				{
					if(!pLight->pModel)
						continue;

					if(auto pMaterial = pLight->pModel->GetMaterial(2))
						pMaterial->SetTexture(3, m_pTextures->Get(L"ShadowBlur" + conv::ToString(count)));
				}
                }

Now there is the line with pEntity-GetComponent<Light>(), which comes from the deferred light plugin. Up until now they've all be in the same project as my engine, so it wasn't a problem. Now that I'm completely seperating them, I've got to come up with a solution. Internally the shadow plugin will require the deferred light plugin to being loaded, so I've basically got three choices I can think of:

- Implicitely link to the light dll in the shadow dll project, export the Light-component class from the dll and use it directly there. My least favourite option, as is produces a quite unclean mess (exporting classes with possible storages of e.g. std::wstring, which I'm using excessively to exchange data in my components, etc...), and requires at least all header files of the first plugin to create another plugin that depends upon it. Not even talking about all the other issue like binary compatibility, etc...

- Have one header file like "Dev.h" for each plugin that possibly supports extentsion, where one/multiple pure-virtual classes are declared. The component and other classes being used will then derive both the engines component base class and this interface. The interface will have specific getter/setter methods for all the components attributes, as well as a method for retrieving the internal choosen id to identify the type of component, when querying a entity e.g. . This approach will still require one header file from the first plugin to create any extention to it. Plus, it will require some unsafe casting, since from the entity I can only return a reference/pointer to the internal component base class, which I'll then have to cast to a pointer to the plugins component interface. I hope that doesn't sound too confusing, thinking about it I'm not even 100% sure if this would work. Plus, it will introduce quite an amount of additional work for developers of a plugin to support extentions, and possibly generate a lot of plugins that aren't extendable because they lack such a "dev" header file.

- Last option, I'll put a special "dll-component" interface (pure virtual class) in the engine. It will have one generic getter and setter, which takes a string as identifier and a void* to some data, and another method for retrieving mentioned id. For standard components like developed inside the engine, this getters and setters are overridden with an emtpy implementation in the base component class. Only in the plugin the dev will need to implement it using e.g. a switch statement for the identifier string to map the hard-coded member attributes like name, size, etc... I'd then have each dll/plugin/module register their components id with a string, so that it can then later easily be accessed. The finished product would look like this:

auto vEntities = m_pEntities->EntitiesWithComponents<ShadowCaster>();

size_t count = 0;
for(auto pEntity : vEntities)
{
    // do other stuff here ...

    if(IComponentDll* pLight = pEntity->GetComponent(ComponentRegistry::GetId(L"Light")))
    {
        gfx::IModel* pModel = dynamic_cast<gfx::IModel*>(pLight->GetAttribute(L"Model")); // maybe some helper-function like GetComponentAttribute<gfx::IModel*>(pLight, L"Model");
        if(!pModel)
            continue;

        if(auto pMaterial = pModel->GetMaterial(2))
            pMaterial->SetTexture(3, m_pTextures->Get(L"ShadowBlur" + conv::ToString(count)));
    }
}

Now this does still appear a little flawed to me. Mostly because it still puts some work to the developer (implementing out the setter/getter), and introduces eigther blind-trust unsafe static or dynamic casting (I don't really like dynamic casting that much). But at least I wouldn't have to export more code, and the interface part would be generic on the engines side.

_____________________________

So, what do you say? Which one of those methods, if any, would you use? If none of them appears good to you, what would you do different? All suggestions are welcomed!

I can't answer your question specifically, but a few ideas here might help:

A standard way of getting around dynamic casts is using a GetInterface method, passing in an interface id. That id can be declared in the interface header file as a static member of the interface structure/class in such a way that you can then use template methods. You can see something similar in the GetInterface members in this file.

As an aside, for performance I'd not use strings but a string hash or id. You can easily convert by adding a function to get the hash/id, use this outside the loop and then use hash/ids inside.

One other note is that if you're iterating through entities getting a component you should likely consider having that component stored in it's own array and then iterating through that.

A standard way of getting around dynamic casts is using a GetInterface method, passing in an interface id. That id can be declared in the interface header file as a static member of the interface structure/class in such a way that you can then use template methods. You can see something similar in the GetInterface members in this file.

Thanks, that sounds intersting indeed, but does it really help in the specific case here? The dynamic case comes from aquiring an attribute via the GetAttribute-method, which might internally look like this:

void* GetAttribute(const std::wstring& stName)
{
       // hashing migh be a good idea after all :D
       if(stName == L"Model")
                return m_pModel; // gfx::IModel* m_pModel;
       else if(stName == L"Name")
                return m_stName; // std::wstring m_stName;
       else
                return nullptr;
}

The problem here is more or less that I need to access an attribute of variable type, from int over std to custom interface. Can such a GetInterface-or similar method be applied here to eliminate this problem, or do I need another solution?

As an aside, for performance I'd not use strings but a string hash or id. You can easily convert by adding a function to get the hash/id, use this outside the loop and then use hash/ids inside.

Thanks, I'm planning on doing so, applying it overall will take some time though, and Nr1 priority so far is to seperate and get to work all modules as dll plugin...

One other note is that if you're iterating through entities getting a component you should likely consider having that component stored in it's own array and then iterating through that.

I'm not so sure about this. For example, here I need to access the light-component too, while originally iterating over the shadow components. I know that some people define it that the components are not equivalent to the entitiy, but things have worked well so far, and from what I get there are a hundred different views on entity/component systems. Still, I agree that some kind of sorting would be helpful, I'll be implementing some sort of cache-system to store buckets of entities based on their components. Unless I run into performance issue, where I'll consider applying a more cache-friendly system, but so far, even thousands of entities with tons of different systems sometimes cross-referencing components haven't had any extraordinary negative performance impact...

The problem here is more or less that I need to access an attribute of variable type, from int over std to custom interface. Can such a GetInterface-or similar method be applied here to eliminate this problem, or do I need another solution?

Yes. Rather than return a void* you return an IInterface* type which has the virtual function for GetInterface defined. This then allows dynamic-cast free casting, made easier via templates. It still costs virtual function call though.

I'm not so sure about this. For example, here I need to access the light-component too, while originally iterating over the shadow components.

You can still access one component whilst iterating through another; you can get the entity for the light and then get the shadow component for that. Using arrays of each component also allows low level systems like renderers to completely avoid using any typing or entity system.

Yes. Rather than return a void* you return an IInterface* type which has the virtual function for GetInterface defined. This then allows dynamic-cast free casting, made easier via templates. It still costs virtual function call though.

Ah, so I'd have to create an IInterface derived class for every type I want to convert to, is this correct? Doesn't that lead to having to distribute the interfaces I want to convert to the end user after all? I think I'm missing some key piece, but wouldn't that then have to be used something like this:

IInterface* pModelInterface = pLight->GetAttribute(L"Model");
gfx::IModel* pModel = pModelInterface->GetInterface<IModelInterface>();
IInterface* pStringInterface = pLight->GetAttribute(L"Name");
const std::wstring& stName = pModelInterface->GetInterface<IStringInterface>();

Does that mean I have to eigther define the interfaces for all common types in the engine as part of the dll interface, or have the interfaces included by the plugins header file (which I'd want to avoid if possible); or am I really missing something? (some explanatory pseudo-code would be nice, in that case though )

You can still access one component whilst iterating through another; you can get the entity for the light and then get the shadow component for that. Using arrays of each component also allows low level systems like renderers to completely avoid using any typing or entity system.

I quess that would work, so I'll consider it a close option in case it is necessary, premature optimization and whatenot (or, lets add, me being to lazy to rework this part yet). As for low level systems, that certainly can be a nice addition, but I've seperated my low level render systems so that they are controlled via a gfx-layer, that is used by the component system, so that this wasn't really an advantage here, so there is no need/function for using anything from the entity system directly anyway ^^

Ah, so I'd have to create an IInterface derived class for every type I want to convert to, is this correct? Doesn't that lead to having to distribute the interfaces I want to convert to the end user after all?

If you want some form of fast type conversion without dynamic cast you'll need an interface type which you derive from, yes. In order for a plugin writer to be able to use code, they need the declaration of that code at minimum - there's little use being able to cast a type to another type if they can't then use that type. So yes you'll need to distribute interfaces for the functionality. An alternative is to write a message passing or signal/slot type system, but this still requires that you define some way for programmers to understand how to use this.

Does that mean I have to eigther define the interfaces for all common types in the engine as part of the dll interface, or have the interfaces included by the plugins header file (which I'd want to avoid if possible); or am I really missing something? (some explanatory pseudo-code would be nice, in that case though )

Not quite sure I'm getting the issue here. If you want someone to be able to use an interface they need the declaration of that interface, which is usually in a header file.

I quess that would work, so I'll consider it a close option in case it is necessary, premature optimization and whatenot (or, lets add, me being to lazy to rework this part yet). As for low level systems, that certainly can be a nice addition, but I've seperated my low level render systems so that they are controlled via a gfx-layer, that is used by the component system, so that this wasn't really an advantage here, so there is no need/function for using anything from the entity system directly anyway ^^

It's worth your while following your current course through to completion if you've already got some way in. The issue I'm referring to is not really a premature optimization, but just a good point to start. There's some good material to be found by searching the web for information on data oriented design (often referred to as DOD). It's also a very good principal to not overly generalize systems until you need to - there's some good information on that in this blog and comment discussion.

Not quite sure I'm getting the issue here. If you want someone to be able to use an interface they need the declaration of that interface, which is usually in a header file.

Well, I want to do the somehow direct opposite, namely use a component without having given the declaration of any specific interface of the plugin that. All the needed interfaces should be given by the engine itself - and it almost is. The BaseComponent-class is part of the engine, and every component is supposed to derive from it. So all that I need now is a somewhat safe method to retrieve specific attributes - if I was to rely on the dynamic_cast, there wouldn't be the need to have any header file for using another plugin, which is my final goal - the problem with requiring a header file is that the developer of the plugin will have to make sure that the plugin can be extendet explicetly - I'd prefer a method that puts most of the work at the engine, with having a clean interface that "simply" has to be implemented in order to allow anyone to develope their plugin on top of the engine and any other plugin, by just knowing what components it declared, and which data members it has. Just to emphasise, this is not about using entirely new classes/interfaces that the plugin defines - its just about working with the eigther POD or engine defined interface data members of the components, as sort of a messaging system.

I hope this helped to clarify a bit, and didn't go toward confusing you about my goals even more (I've got a real hard time expressing this, probably due to my limitations in the english language...). Does it appear more clear now?

The issue I'm referring to is not really a premature optimization, but just a good point to start.

My bad, I wasn't referring to your advice as premature optimization, but to the act of applying them for the reason I mentioned, as in cache-friendlyness, since other than that I'm pleased with the system as it (not meaning that I don't see any possibilty, or don't want to improve, but not right now).

It's also a very good principal to not overly generalize systems until you need to - there's some good information on that in this blog and comment discussion.

I do agree in matter of productivity especially in buisness, as in the article, but for this project, its not really any matter for me. Up until the next university semester starts, I'm pretty much coding the engine without any clear project, so up until then I just implement features I've got on my todo-list that will make further developement probably easier. Pretty much the whole game engine up until now was unnecessary for the one simple tower defense game I've developed until now, its a whole "future-paced" project if you will. Even if it would never be used in any other game project, unlike the future focused guy in the article, I'd still have enjoyed the time I spent developing, and it has kept me focused on the project for almost a year now without any major break, while normally I'm loosing interesting in hobby projects after a few weeks/months. Not that I don't appreciate the additional input, just to state the I'm well aware what I'm doing and why I'm doing it ;)

Ignoring all the work up on plugin structure, it seems you are focused on generic attribute access. Focusing just on that feature, then the way I've always looked at this is a getter/setter solution either based on variants or by templates which do all the type checking. Using variants is the easiest solution though it requires a bit of dynamic memory for some types. Basically the variant return is:

struct Variant
{
  enum
  {
     kSInt, kUInt, kFloat, kDouble, etc...
  };
  uint32_t    Type;
  union
  {
    int32_t   SInt;
    uint32_t  UInt;
    etc
  };    
};

Now, you can retrieve the attributes using a "bool GetAttr( const std::string& attr, Variant& outVar )" generic interface. The bool result tells you if the attribute exists and then when you cast the variant to some type the accessors (the struct would be all private with cast helpers) would validate that the internal data casts to the type you request.

Using the same concept though, you can embed the variant like checks into the generic getter such as: "bool GetAttr( const std::string& attr, uint32_t typeKey, void* outData )". Again, you have the bool result saying the attribute exists or not but now you pass the type into the source and check the desired type is valid for the attribute in question. Internally you simple implement something like:

switch( typeKey )
{
  case Type::SInt:
    if( attr=="SomeVal" )
      *(reinterpret_cast< int32_t* >( outData )) = mSomeVal;
      return true;

It gives some small amount of type safety though of course people can mismatch the type key and the output value and cause all sorts of problem. You can also turn the entire thing into a table based solution such that you would write out:

  { Type::SInt, "SomeVal", offsetof( SomeClass, mSomeVal ) }

Then on startup build a nice hash table of the different types which should be about as fast as the switch/case stuff without as many likely typo's involved.

Finally, with the combined solution you can write templates such as:

template< typename _Type >
bool GetAttr( const std::string& name, _Type& outData )
{
  assert( false ); // Non-specialized type..  Use generic access.
  return false;
}

template<>
void GetAttr( const std::string& name, uint32_t& outData )
{
  return GetAttr( name, Type::UInt, &outData );
}

Overall the variant is the safest variation and best C++ style in the general sense but it comes at a performance/memory bandwidth cost compared to the less safe more "C" style accessor.


Now, with all the above, there is one other item to keep in mind when moving to the DLL solution. Memory allocations. I assume you either know about this or have worked around it. Basically unless all the plugins and the executable share a common copy of msvcrt<xxx>.dll, the memory will come from different memory managers and passing the memory around is unsafe. Even using the std::string stuff is unsafe since the memory comes from different allocators. In the past I simply wrote a dll which I called "Core" and linked that against the runtimes, then my executable and all plugins had to link against that to get the common allocator. It is not the best solution by far since then everyone has to use the same compiler, or at least the same version of msvcrt, but on the other hand, linking up shared memory management on Windows is a pure hell to get correct. Otherwise, you need to make absolutely sure you never pass memory back and forth between plugins without a dedicated "alloc/free" pair of functions in each plugin. I.e. even the most simple: "GetName()" would need an equivalent "FreeName( p )" without some solution to the memory problem.

Just another gotcha to worry about.

Well, I want to do the somehow direct opposite, namely use a component without having given the declaration of any specific interface of the plugin that. All the needed interfaces should be given by the engine itself - and it almost is. The BaseComponent-class is part of the engine, and every component is supposed to derive from it. So all that I need now is a somewhat safe method to retrieve specific attributes - if I was to rely on the dynamic_cast, there wouldn't be the need to have any header file for using another plugin, which is my final goal

You cannot dynamic_cast to a type which has not been fully defined, so the limitations are the same as for the GetInterface system. Note that you each plugin does not need to have it's own interface, they can implement interfaces you have defined in your engine.

@AllEightUp:

Hm, that sounds like it, but also complicated again, having to define lots of stuff for each possible data type, which can get quite enormous (theoretically, I'd have to define each primitive type, engine class and possible common std::classes). I quess the safety comes at a cost, but I didn't expect that it would need so much extra work per type... I thought there'd be a more general solution. Well, browsing through the web I've found kind of a universal solution, still requiring RTTI but appearantly way faster, which I could apply in a helper function. I'm talking about using typeid. Since I only care about that the types are equal, wouldn't it fare just as well if I just did that?

class BaseComponent
{
public:
	template<typename Type>
	Type* GetAttributeSafe(const std::wstring& stName)
	{
		std::type_info inType;
		if(auto pAttribute = GetAttribute(stName, &inType))
		{
			if(inType == typeid(Type))
				return (Type*)pAttribute;
		}

		return nullptr;
	}

protected:

	virtual void* GetAttribute(const std::wstring& stName, std::type_info& outType) const = 0;
}

class SpecificComponent: BaseComponent
{
private:
	void* GetAttribute(const std::wstring& stName, std::type_info& outType)
	{
		if(stName == L"Model")
		{
			 outType = typeid(m_pModel);
			 return m_pModel;
		}
		
		return nullptr;
	}

	gfx::IModel* m_pModel;
}

// usage:
gfx::IModel* pModel = pLight->GetAttribute<gfx::IModel>(L"Model"); // returns nullptr if eigther attribute doesn't exist or types doesn't match

Thats kind of like what you suggested, just with using RTTI instead of making up a custom system, and thus not having to do a custom . Does that sound reasonable, or are there any drawbacks I'm not seeing right now?

Now, with all the above, there is one other item to keep in mind when moving to the DLL solution. Memory allocations. I assume you either know about this or have worked around it. Basically unless all the plugins and the executable share a common copy of msvcrt.dll, the memory will come from different memory managers and passing the memory around is unsafe. Even using the std::string stuff is unsafe since the memory comes from different allocators. In the past I simply wrote a dll which I called "Core" and linked that against the runtimes, then my executable and all plugins had to link against that to get the common allocator. It is not the best solution by far since then everyone has to use the same compiler, or at least the same version of msvcrt, but on the other hand, linking up shared memory management on Windows is a pure hell to get correct. Otherwise, you need to make absolutely sure you never pass memory back and forth between plugins without a dedicated "alloc/free" pair of functions in each plugin. I.e. even the most simple: "GetName()" would need an equivalent "FreeName( p )" without some solution to the memory problem.

I've definately heard of it, thats why I tried to keep as much as possible internal in my plugins. But what I've also heard so far is that using pointer/references is somewhat safe, like that a std::wstring& will work cross-compiler, since its a pointer and the size will be the same regardless, while only the internal "layout" of the wstring will differ... am I missinformed? Plus, right now my whole engine is a dll that both the plugins and the application have to link against, that seems to be just what you are talking about, isn't it? By god, I don't want to write a dealloc-function for every simply getter I export :/

You cannot dynamic_cast to a type which has not been fully defined, so the limitations are the same as for the GetInterface system. Note that you each plugin does not need to have it's own interface, they can implement interfaces you have defined in your engine.

Hmm, thats not really what I wanted to imply, the only real "problem" I had with the GetInterface solution (as well as AllEightUps) was the I required to declare a new interface with custom id etc.. for each type that was possibly going to be used, which at least in my had made it seem a little much work, where I'd prefered a more generalized method. In that matter, what are your thoughts on the method using typeid I proposed before?

Oh, and refering back to your comment about DOD, I've actually just now realized that my low level render system is in fact data driven, I'm employing a render queue system based on render commands processed in such a linear fassion as mentioned in the presentation, quess I'm not to far offtrack. Just as a quick note. Its strange to see that you are already using a "pattern" without really knowing its name xD