Jump to content
  • Advertisement
  • 11/19/14 10:47 PM
    Sign in to follow this  

    Implementing a Meta System in C++

    General and Gameplay Programming

    occash
    • Posted By occash
    Hi everybody! I am relatively new to game development, however I would like to share my experience in developing a meta system for C++. I faced the problem of embedding a scripting language when I was developing my own 3D game engine. There can be many solutions for embedding a specific language (for example, Luabind for Lua and boost.Python for Python). Having such a variety of tools, one obviously should not reinvent the wheel. I started by embedding the simple and fast Lua programming language with the Luabind library. I think it is very good, you may wish to see yourself: class_("BaseComponent") .def(constructor<>()) .def("start", &BaseScript::start, &ScriptComponentWrapper::default_start) .def("update", &BaseScript::update, &ScriptComponentWrapper::default_update) .def("stop", &BaseScript::stop, &ScriptComponentWrapper::default_stop) .property("camera", &BaseScript::getCamera) .property("light", &BaseScript::getLight) .property("material", &BaseScript::getMaterial) .property("meshFilter", &BaseScript::getMeshFilter) .property("renderer", &BaseScript::getRenderer) .property("transform", &BaseScript::getTransform) This piece of code looks highly readable to me. Class registration is simple, at least I see no obstacles. However, this solution is for Lua only. Inspired by the Unity script system, I decided to add support for several languages into the engine, as well as a platform for interaction between them. Yet such tools as Luabind are not quite suitable for these: most of them are built on C++ templates and generate code only for a pre-specified language. Each class must be registered in each of the systems. Any user of a system has to manually define template instantiations of every class for every scripting language. It would be great to have just one database for all script engines. Moreover, it would be nice to have the ability to load a type's specifications from plugins within runtime. Binding libraries are not good for this - it must be a real metasystem! I could see no way for adopting an existing solution. Existing libraries turned out to be huge and awkward. Some seemingly smart solutions have additional dependencies or require special tools such as Qt moc and gccxml. Of course, one could find good alternatives, such as the Camp reflection library. It looks like Luabind: camp::Class::declare("MyClass") // ***** constant value ***** .function("f0", &MyClass::f).callable(false) .function("f1", &MyClass::f).callable(true) // ***** function ***** .function("f2", &MyClass::f).callable(&MyClass::b1) .function("f3", &MyClass::f).callable(&MyClass::b2) .function("f4", &MyClass::f).callable(boost::bind(&MyClass::b1, _1)) .function("f5", &MyClass::f).callable(&MyClass::m_b) .function("f6", &MyClass::f).callable(boost::function(&MyClass::m_b)); However, the performances of such solutions leave much to be desired. Therefore, I decided to develop my own metasystem, as any normal programmer would, I think. This is why the uMOF library has been developed.

    Meet the uMOF

    uMOF is a cross-platform open source library for meta programming. It resembles Qt, but, unlike Qt, it is developed by using templates. Qt declined the use of templates due to syntax matters. Although their approach brings high speed and safe memory use, it requires the use of an external tool (MOC compiler) which is not always convenient. Now let's get down to business. To make meta information available to users in objects inherited from Object class, you should write OBJECT macro in the class definition. Now you can write EXPOSE and PROPERTIES macros to define functions and properties. Take a look at this example: class Test : public Object { OBJECT(Test, Object) EXPOSE(Test, METHOD(func), METHOD(null), METHOD(test) ) public: Test() = default; float func(float a, float b) { return a + b; } int null() { return 0; } void test() { std::cout << "test" << std::endl; } }; Test t; Method m = t.api()->method("func(int,int)"); int i = any_cast(m.invoke(&t, args)); Any res = Api::invoke(&t, "func", {5.0f, "6.0"}); In the current version, insertion of meta information is invasive; yet development of an external description is in progress. Due to the use of advanced templates, uMOF is very fast and compact. A downside is that not all compilers are supported because of new C++11 features utilized (for example, to compile on Windows, you would need the latest version of Visual C++, the November CTP). Since usage of templates may not be too pleasant for some developers, they are wrapped up in macros. This is why the public API looks rather neat. To prove my point, here are benchmark test results.

    Test results

    I compared meta-systems over three parameters: (a) compilation and link time, (b) executable size and (c) function call time. I took a native function call as the reference. All systems were tested on a Windows platform with Visual C++ compiler. These results visualized:
    gistogram1.png gistogram2.png gistogram3.png
    I also considered testing other libraries:
    • Boost.Mirror
    • XcppRefl
    • Reflex
    • XRtti
    However, currently this appears impossible because of various reasons. The Boost.Mirror and XcppRefl look promising, but they are not yet in an active development stage. Reflex needs GCCXML tool, but I failed to find any adequate substitution of that for Windows. Xrtti does not support Windows either in the current release.

    What is in the pipeline?

    So, how does it work? Variadic templates and templates with functions as arguments give speed and a compact binary. All meta information is organized as a set of static tables. No additional actions are required at runtime. A simple structure of pointer tables keeps binary tight. Find an example of function description below: template struct Invoker { typedef Return(Class::*Fun)(Args...); inline static int argCount() { return sizeof...(Args); } inline static const TypeTable **types() { static const TypeTable *staticTypes[] = { Table::get(), getTable()... }; return staticTypes; } template inline static Any invoke(Object *obj, F f, const Any *args, unpack::indices) { return (static_cast(obj)->*f)(any_cast(args[Is])...); } template static Any invoke(Object *obj, int argc, const Any *args) { if (argc != sizeof...(Args)) throw std::runtime_error("Bad argument count"); return invoke(obj, fun, args, unpack::indices_gen()); } }; The Any class plays an important role in the library performance. It allows allocating memory for instances and stores the associated type information efficiently. I used hold_any class from the boost.spirit library as a reference. Boost also uses templates to wrap types. Types, which are smaller than a pointer, are stored in void* directly. For a bigger type, the pointer refers to an instance of the type. template struct AnyHelper { typedef Bool::value> is_pointer; typedef typename CheckType::type T_no_cv; inline static void clone(const T **src, void **dest) { new (dest)T(*reinterpret_cast(src)); } }; template struct AnyHelper { typedef Bool::value> is_pointer; typedef typename CheckType::type T_no_cv; inline static void clone(const T **src, void **dest) { *dest = new T(**src); } }; template Any::Any(T const& x) : _table(Table::get()), _object(nullptr) { const T *src = &x; AnyHelper::is_small>::clone(&src, &_object); } I had to reject using RTTI - it is too slow. Types are checked only by comparison of pointers to the static tables. All type modifiers are omitted, so that, for example, int and const int are treated as the same type. template inline T* any_cast(Any* operand) { if (operand && operand->_table == Table::get()) return AnyHelper::is_small>::cast(&operand->_object); return nullptr; }

    How to use the library?

    Script engine building becomes simple and nice. For example, it is enough to define an generic call function for Lua. It will check the number of arguments and their types and, of course, call the function itself. Binding is also not difficult: just save MetaMethod in upvalue for each function in Lua. All objects in uMof are "thin", that is to say they only wrap around pointers referring to records in the static table. Therefore, you can copy them without worrying about the performance. Find an example of Lua binding below: #include #include #include #include class Test : public Object { OBJECT(Test, Object) EXPOSE( METHOD(sum), METHOD(mul) ) public: static double sum(double a, double b) { return a + b; } static double mul(double a, double b) { return a * b; } }; int genericCall(lua_State *L) { Method *m = (Method *)lua_touserdata(L, lua_upvalueindex(1)); assert(m); // Retrieve the argument count from Lua int argCount = lua_gettop(L); if (m->parameterCount() != argCount) { lua_pushstring(L, "Wrong number of args!"); lua_error(L); } Any *args = new Any[argCount]; for (int i = 0; i < argCount; ++i) { int ltype = lua_type(L, i + 1); switch (ltype) { case LUA_TNUMBER: args.reset(luaL_checknumber(L, i + 1)); break; case LUA_TUSERDATA: args = *(Any*)luaL_checkudata(L, i + 1, "Any"); break; default: break; } } Any res = m->invoke(nullptr, argCount, args); double d = any_cast(res); if (!m->returnType().valid()) return 0; return 0; } void bindMethod(lua_State *L, const Api *api, int index) { Method m = api->method(index); luaL_getmetatable(L, api->name()); // 1 lua_pushstring(L, m.name()); // 2 Method *luam = (Method *)lua_newuserdata(L, sizeof(Method)); // 3 *luam = m; lua_pushcclosure(L, genericCall, 1); lua_settable(L, -3); // 1[2] = 3 lua_settop(L, 0); } void bindApi(lua_State *L, const Api *api) { luaL_newmetatable(L, api->name()); // 1 // Set the "__index" metamethod of the table lua_pushstring(L, "__index"); // 2 lua_pushvalue(L, -2); // 3 lua_settable(L, -3); // 1[2] = 3 lua_setglobal(L, api->name()); lua_settop(L, 0); for (int i = 0; i < api->methodCount(); i++) bindMethod(L, api, i); } int main(int argc, char *argv[]) { lua_State *L = luaL_newstate(); luaL_openlibs(L); bindApi(L, Test::classApi()); int erred = luaL_dofile(L, "test.lua"); if (erred) std::cout << "Lua error: " << luaL_checkstring(L, -1) << std::endl; lua_close(L); return 0; }

    Conclusion

    Let us summarize what we have got. uMOF advantages:
    • Compact
    • Fast
    • No external tools, just a modern compiler needed
    uMOF disadvantages:
    • Supported by only modern compilers
    • Auxiliary macros are not quite polished
    The library is in a rather raw stage yet. However, the approach leads to good results. So I'm going to implement a few useful features, such as variable arity functions (default parameters), external description of meta types and property change signals. Thank you for your interest. You can find the project here - https://bitbucket.org/occash/umof. Comments and suggestions are welcome, as "comments" I suppose.


      Report Article
    Sign in to follow this  


    User Feedback


    That looks very nice.

     

    I too dread the use of external libraries (luabind drags in boost, others require pre processing steps)

    Does is also allow binding of free floating (non member) functions?

    Share this comment


    Link to comment
    Share on other sites

    Great article. I see that you implementing meta structures in header file (OBJECT, EXPOSE macros). Maybe I am wrong but I think that this approach will increase executable size as the project gets bigger (compiler will generate multiple meta structures for the same type in each compiled cpp file). Have you tested executable size with uMOF on bigger project? Btw. the Bitbucket link doesn't work.

    Share this comment


    Link to comment
    Share on other sites
    Thank you all for your positive comments.
     
    Endurion,
    Sure, you can bind free functions as well as static methods.
     
    majo33,
    Sorry, link is correct now.

     

    These macros will not increase the executable. This works because I define the static array inside a method. And for methods, the linker does recognize multiple identical definitions and throws away the copies.

    Share this comment


    Link to comment
    Share on other sites

    for example, to compile on Windows, you would need the latest version of Visual C++, the November CTP

     

    You should avoid saying things like this in order to future-proof your article. In a year, this statement won't be true. Prefer to list the specific version of VS required or, ideally, list the specific C++11 features that you are using that you know have compatibility issues.

     

    I compared meta-systems over three parameters: (a) compilation and link time, (b) executable size and © function call time. I took a native function call as the reference. All systems were tested on a Windows platform with Visual C++ compiler.

     

     Good plan, but realistically all those graphs are useless at best (and misleading at worst) without more details about what the specifics of the benchmarks are.

     

    It would be particularly useful to know what the cost, per-object, of describing an object, field and method using your macro system is.

     

    Reflex needs GCCXML tool, but I failed to find any adequate substitution of that for Windows

     

    GCCXML can actually be compiled for Windows with Visual Studio relatively easily, but you do have to fix a few syntax errors (last I checked).

     

    All type modifiers are omitted, so that, for example, int and const int are treated as the same type.

     

    This implies that a system accessing a const field via your reflection interface can't know if that field is const, and thus can break a class invariant, no?

     

    Overall, this looks like a decent piece. I'd like to see it expanded with more information about the techniques you're actually using to implement the description of the types (the macros and their underlying templates); I find this is often the most interesting aspect to people learning about building these systems. I don't think this is a serious enough issue to mark the article for rejection, but if you added it I'd definitely mark it for acceptance.

     

    (Also, as an aside, I'd consider prefixing your API macros with UMOF_ or something -- OBJECT in particular is very common and likely to cause a name collision.)

    Share this comment


    Link to comment
    Share on other sites

    Looks really cool!

     

    On the bitbucket page, in the 'usage' code example, you have:

    OVERLOAD(lol, Test, float, float, float)

    What is 'lol'? I don't see it defined anywhere in that example.

     

    Also, to me personally, {Test, float, float, float} looks like 'Test' is taking three floats, but the first one is actually the return value.

    It may be clearer to users if you changed the order to <return> <funcname> <args...> like C and C++.

    OVERLOAD(lol, float, Test, float, float) //Example reordering

    I second the request for you to prefix your macros with UMOF_, or even just a lowercase 'u':

    uOBJECT(Test, Object)
    uEXPOSE(Test, 
        uMETHOD(func),
        uMETHOD(null),
        uMETHOD(test)
    )

    That'll definitely help reduce namespace conflicts.

     

    Speaking of namespaces, does UMOF support exposing namespaces to scripting languages?

     

    What about globals - is UMOF's job to define and expose the functions and class types, or does it also expose instances? Do I expose globals to Lua's environment using Lua directly, or do I do it through UMOF as an intermediary?

    Share this comment


    Link to comment
    Share on other sites

    Also, to me personally, {Test, float, float, float} looks like 'Test' is taking three floats, but the first one is actually the return value.


    His code is following common idioms for C++-based reflection/binding libraries. I wouldn't change it like that. I might change it to take a signature instead of a varying argument list, though, e.g.

    OVERLOAD(lol, Test, float(float, float));

    Share this comment


    Link to comment
    Share on other sites

    Hi, how do you solve the problems with the Garbage Collector en Lua? (low performance while GB is cleaning)

     

    Thank you!

    Share this comment


    Link to comment
    Share on other sites

    Hi, Artem.

     

    Liked your lib. 

    I've created a sane Makefile and Code::Blocks project for it ( to build shared libs ).

    Because for me your premake scripts didn't work at all ( using premake from Linux Mint 17 repos )

    + IMHO whole premake thing just sucks a little ( taking in account that you have not as much files in your src/ dir - why use build system generator at all ? ).

     

    Would you be so kind to include something like that?

     

    Anyways - great job and neat article.

    My fork with Makefiles and Code::Blocks project is available at bitbucket

    This should work with Linux/Windows, however, one will not be able to build test benchmark and will be forced to use premake.

    Share this comment


    Link to comment
    Share on other sites
    Thank you all for comments!
     
    Josh Petrie,
    Thank you for your suggestions on how to improve the article. I will try to modify article and code accordingly.
    I have only a few remarks.
    One still have to build GCCXML and resolve all errors, what can be annoying. uMOF is designed to be independant from external tools.
    Type modifiers only omitted for utility class Any. This doesn't break a class invariant.
     
    Servant of the Lord,
    Thank you for your remarks. I have corrected the readme on bitbucket page.
    Certainly I will add the prefix.
    Unfortunately uMOF can't handle namespaces and globals. uMOF exposes classes, not instances.
     
    SeanMiddleditch,
    Thank you, I really like your solution!
     
    augus1990,
    The question is still yet to be answered. uMOF can't help with garbage collection.
     
    syskrank,
    Thank you.
    I agree that premake is unnesesary to build such a small project. I will add makefile for linux users.

    Share this comment


    Link to comment
    Share on other sites


    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!