Should I use a Lua wrapper class in my engine?

Just depends on what you're comfortable with. Some people choose to use a library like tolua++ rather than use Lua directly, on the other hand, in the Bitsquid engine they do what you're doing, and argue that their engine's Lua API is cleaner as a result.
i.e. Sometimes a binding layer like tolua++ will make it easy to write code that makes sense in C++ but is messy in Lua, or code that is inefficient and not idiomatic, etc...

I made my own very lightweight wrapper, which requires you to list all the class members to be bound on the C++ side using magic macros, and then creates table on the Lua side containing the member functions. C++ classes aren't turned into Lua classes by this binding layer (and aren't garbage collected by default either) -- instead, Lua code that's calling C++ code has to use a kind-of C-style of OOP that looks like: Class.Function( this, arguments )

e.g.
C++ class, and a (non-bound) structure to be passed around:

struct Stuff { int value; };
class Test
{
public:
	eiBind(Test);//make the binding system a friend, so it can access private variables
	Stuff DoStuff(int i, Test* nil, const char* text)
	{
		eiASSERT( nil == NULL );
		printf("DoStuff %s", text);
		Stuff s = {i};
		return s;
	}
	void   PrintStuff1(Stuff  s) { printf("1: %d\n", s.value); }
	Stuff* PrintStuff2(Stuff* s) { printf("2: %d\n", s->value); return s; }
	void   PrintStuff3(Stuff& s) { printf("3: %d\n", s.value); }
private:
	int m_stuff;
};

//Manual reflection system:
eiBindClass( Test )
	eiLuaBind( LuaCall ) // Make the methods callable from Lua
	eiBeginMethods()
		eiBindMethod( DoStuff )
		eiBindMethod( PrintStuff1 )
		eiBindMethod( PrintStuff2 )
		eiBindMethod( PrintStuff3 )
	eiEndMethods()
eiEndBind();

Optional Lua - extend the C++ 'Test' class with a Lua 'LuaTest' class:

LuaTest = LuaTest or {}
LuaTest.__index = LuaTest
function LuaTest:DoStuff()
	Test.DoStuff(self.member, 42, nil, 'text')
end
function NewLuaTest(x)
	local object = {}
	setmetatable(object, LuaTest)
	object.member = x
	return object
end

Lua - code to use the above C++ and Lua classes:

function init( test ) -- 'test' is a C++ 'Test*'
	--test the C++ version:
	local stuff = Test.DoStuff(test, 42, nil, 'text')
	Test.PrintStuff1(test, stuff)
	local s2 = Test.PrintStuff2(test, stuff)
	Test.PrintStuff3(test, s2)

	--test the Lua wrapper version:
	local object = NewLuaTest(test)
	object:DoStuff()
	return test
end

I find writing manual Lua bindings to be quite tedious, so I prefer to have a library like this that can automatically make my C++ code callable from Lua.

Thanks for the reply Hodgman!

I made my own very lightweight wrapper, which requires you to list all the class members to be bound on the C++ side using magic macros, and then creates table on the Lua side containing the member functions. C++ classes aren't turned into Lua classes by this binding layer (and aren't garbage collected by default either) -- instead, Lua code that's calling C++ code has to use a kind-of C-style of OOP that looks like: Class.Function( this, arguments )

Having a few macros like those to speed class binding looks really useful. I'll try to implement something like that while still allowing manual binding for special cases.

Yeah, we had a home-brewed C++/Lua binding system at the last games comapany I worked for, and you could either use macros like in mine, or they had a special 'manual' macro, which meant that the function should be bound, but that it just takes a lua_State* and you promise to fetch the arguments / push the returns by yourself.

If you want to peek under the hood of my system (beware, ugly/confusing code, and not very battle tested yet!), my macros are in bind.h and the lua binding glue is in bindlua.h.

For more battle-tested versions, check out any on this list, but any C++ binding system is likely to involve a fair bit of C++ black magic...

That's some TMP black magic indeed...

Are your macros able to reflect the functions arguments list? How is it done? Like knowing how many arguments a function needs and the types of the arguments.

If you can use LuaJIT, you might find it easier to write C wrappers for all the class functions and invoke them through the FFI.

If want to avoid all the C wrappers, the FFI can bind directly to the class members (see this), its a bit brittle though, however, if you are targeting only a single compiler+platform, its trivial* to generate a script to encode the symbols and bind everything in the FFI. Not only is that faster (dev wise), and leads to less clutter, but you get LuaJIT with all its awesome features :) (if you factor your code properly, you can even just parse a C or C++ header directly in Lua, meaning you only need to update a singular point).

I went the C++ template and macro magic route (not as heavily as Hodgman), found it horrible to work with (and by "work with" I just mean adding of new functions, enums, classes/objects etc) in the end, been meaning to try the above using the FFI (I use LuaJIT already), unfortunately I haven't had the time yet...

*the cdef generation part, decoration of symbols might be a little more tricky, esp. with MSVC

been meaning to try the above using the FFI (I use LuaJIT already), unfortunately I haven't had the time yet..

Same here! FFI looks very promising, but I've not got around to testing it. I've been a bit scared off how it's C parsing is implemented (basically running a C compiler behind the scenes)... but if you can get it to bind to manually specified symbols, then that might be handy.

Are your macros able to reflect the functions arguments list? How is it done? Like knowing how many arguments a function needs and the types of the arguments.

Yes, they detect the arguments and return types with template magic. This is out of order and simplified slightly, but hopefully should answer "how is it done"

Start off with some typedefs -- memfuncptr is basically going to act like a void* for member functions -- we can use it to store member-function-pointer values while forgetting their actual types.

FnGeneric and FnLua are the final/produced types of the binding systems -- we want to be able to take any member function and convert it into a FnGeneric* (which lets us call it given a binary blob of input arguments and return output buffer - a kind of generic binding system), or into a FnLua* (which lets us call it with args popped from, and return value pushed to a Lua VM - a Lua-specific binding system).

class class_ {};
typedef void (class_::*memfuncptr)(void*);
typedef void (FnGeneric)( void* obj, void* args, void* output );
typedef int (FnLua)( lua_State* );

As you can see on lines #1/2, CallWrapper/LuaCallWrapper aren't implemented yet, but here's how we'd perform the conversion from member function to the above types:

template<class T, memfuncptr fn, class F> void        CallWrapper(void* user, void* argBlob, void* outBlob); //#1
template<class T, memfuncptr fn, class F> int         LuaCallWrapper(lua_State* luaState);                   //#2

template<class T, memfuncptr fn, class F> FnGeneric*  GetCallWrapper(   F f=0) { return &CallWrapper<   T,fn,F>; }
template<class T, memfuncptr fn, class F> FnLua*      GetLuaCallWrapper(F f=0) { return &LuaCallWrapper<T,fn,F>; }

//This bit goes inside your magic macro:
//'T' = a class, 'a' = a function name. e.g. 'T' might be 'std::vector<int>' and 'a' might be 'push_back'.
FnGeneric* call = GetCallWrapper<T,(memfuncptr)&T::a>(&T::a);
FnLua* luaCall = GetLuaCallWrapper<T,(memfuncptr)&T::a>(&T::a);

The 'Get' functions are needed to convert the value "&T::a" into the type of that value, "F". N.B. this is a pre-C++11 trick.

When you call Get*Wrapper the CallWrapper/LuaCallWrapper template is given three template arguments -- the class type that owns the member-function (T), a typeless value of the member function-pointer (fn), and the actual type of the member function pointer (F).

These three template arguments are then used to implement the CallWrapper/LuaCallWrapper function like this:

template<class T, memfuncptr fn, class F>
void CallWrapper(void* user, void* argBlob, void* outBlob)
{
//Given a function, create structures that can hold the arguments and the return value
	typedef ArgFromSig<F>::Storage Args;
	typedef ReturnType<ArgFromSig<F>::Return> Return;
	Args aDefault = {};
	Return rDefault;
//Typecast the input pointers to the above types
	Args& a = argBlob ? *((Args*)argBlob) : aDefault;
	Return& r = outBlob ? *((Return*)outBlob) : rDefault;
//Now call the function, using the above structs as the inputs/output
	Call( r, a.t, (T*)user, (F)fn ); //cast fn back to F to recover it's type
}

template<class T, memfuncptr fn, class F>
int LuaCallWrapper(lua_State* L)
{
//Given a function, create structures that can hold the arguments and the return value
	typedef ArgFromSig<F>::Storage::Type ArgTuple;
	typedef ArgFromSig<F>::Return RawReturn;
	typedef ReturnType<RawReturn> Return;
	int numArgs = lua_gettop(L);
	if( ArgTuple::length != numArgs-1 )
		luaL_error(L, "Expected %d args, received %d", ArgTuple::length, numArgs );
	T* user = internal::CheckUserDataToPointer<T,false>(L, 1, lua_type(L, 1));
	luaL_argcheck(L, user, 1, "NULL reference!");
//pop each argument from the Lua stack into our structure
	ArgTuple a;
	Return r;
	a.ForEach( internal::FetchArgsCall(L, 2) );
//Now call the function, using the above structs as the inputs/output
	Call( r, a, user, (F)fn );
//Push the return value back to the Lua stack
	return internal::PushValue<RawReturn>(L, r.value);
}

The remaining magic is in the Call, ArgFromSig and ReturnType templates.
Firstly though, I use tuples for the args, which don't exist in pre-C++-11, so I implemented them like this (up to 5 values per tuple, just extend the pattern if you need more):

struct Nil
{
	bool operator==( const Nil& ) { return true; }
};

template<class A=Nil, class B=Nil, class C=Nil, class D=Nil, class E=Nil>
                                             struct Tuple                      { A a; B b; C c; D d; E e; template<class F> void ForEach(F& f) { f(a); f(b); f(c); f(d); f(e); } template<class F> void ForEach(F& f) const { f(a); f(b); f(c); f(d); f(e); } const static uint length = 5; typedef A   A; typedef B   B; typedef C   C; typedef D   D; typedef E   E; };
template<class A, class B, class C, class D> struct Tuple<  A,  B,  C,  D,Nil> { A a; B b; C c; D d;      template<class F> void ForEach(F& f) { f(a); f(b); f(c); f(d);       } template<class F> void ForEach(F& f) const { f(a); f(b); f(c); f(d);       } const static uint length = 4; typedef A   A; typedef B   B; typedef C   C; typedef D   D; typedef Nil E; };
template<class A, class B, class C>          struct Tuple<  A,  B,  C,Nil,Nil> { A a; B b; C c;           template<class F> void ForEach(F& f) { f(a); f(b); f(c);             } template<class F> void ForEach(F& f) const { f(a); f(b); f(c);             } const static uint length = 3; typedef A   A; typedef B   B; typedef C   C; typedef Nil D; typedef Nil E; };
template<class A, class B>                   struct Tuple<  A,  B,Nil,Nil,Nil> { A a; B b;                template<class F> void ForEach(F& f) { f(a); f(b);                   } template<class F> void ForEach(F& f) const { f(a); f(b);                   } const static uint length = 2; typedef A   A; typedef B   B; typedef Nil C; typedef Nil D; typedef Nil E; };
template<class A>                            struct Tuple<  A,Nil,Nil,Nil,Nil> { A a;                     template<class F> void ForEach(F& f) { f(a);                         } template<class F> void ForEach(F& f) const { f(a);                         } const static uint length = 1; typedef A   A; typedef Nil B; typedef Nil C; typedef Nil D; typedef Nil E; };
template<>                                   struct Tuple<Nil,Nil,Nil,Nil,Nil> {                          template<class F> void ForEach(F& f) {                               } template<class F> void ForEach(F& f) const {                               } const static uint length = 0; typedef Nil A; typedef Nil B; typedef Nil C; typedef Nil D; typedef Nil E; };

ArgFromSig can then be passed a member-function-pointer type, and can extract the return type into a typedef, and the arg types into a typedef of a tuple.

template<class F> struct ArgFromSig;
template<class R, class T>                            struct ArgFromSig<R(T::*)(void )> { typedef ArgStore<R,Nil,Nil,Nil> Storage; typedef R Return; };
template<class R, class T, class A>                   struct ArgFromSig<R(T::*)(A    )> { typedef ArgStore<R,  A,Nil,Nil> Storage; typedef R Return; typedef A A; };
template<class R, class T, class A, class B>          struct ArgFromSig<R(T::*)(A,B  )> { typedef ArgStore<R,  A,  B,Nil> Storage; typedef R Return; typedef A A; typedef B B; };
template<class R, class T, class A, class B, class C> struct ArgFromSig<R(T::*)(A,B,C)> { typedef ArgStore<R,  A,  B,  C> Storage; typedef R Return; typedef A A; typedef B B; typedef C C; };

//ArgStore is just a wrapper around Tuple that adds some reflection capabilities:
const static uint MaxArgs = 3;
struct ArgHeader { void (*info)(uint& n, uint o[MaxArgs], uint s[MaxArgs]); };
template<class R, class A=Nil, class B=Nil, class C=Nil>
                                    struct ArgStore                { ArgHeader b; typedef Tuple<A, B, C> Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length; o[0]=offsetof(Type,a); o[1]=offsetof(Type,b); o[2]=offsetof(Type,c); s[0]=sizeof(A); s[1]=sizeof(B); s[2]=sizeof(C); } };
template<class R, class A, class B> struct ArgStore<R,  A,  B,Nil> { ArgHeader b; typedef Tuple<A, B>    Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length; o[0]=offsetof(Type,a); o[1]=offsetof(Type,b);                        s[0]=sizeof(A); s[1]=sizeof(B);                 } };     
template<class R, class A>          struct ArgStore<R,  A,Nil,Nil> { ArgHeader b; typedef Tuple       Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length; o[0]=offsetof(Type,a);                                               s[0]=sizeof(A);                                 } };                 
template<class R>                   struct ArgStore<R,Nil,Nil,Nil> { ArgHeader b; typedef Tuple<>        Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length;                                                                                                                      } };					   

ReturnType is only needed to deal with void return types, which aren't values. ReturnType can be copied like a value, even though void cannot, which simplifies things.

template<class T> struct ReturnType       { typedef T   Type; static uint Size() { return sizeof(T); } T value; operator T&() { return value; } void operator=(const T& i){value = i;}};
template<>        struct ReturnType<void> { typedef Nil Type; static uint Size() { return 0;         } Nil value; };

The call template then unpacks the argument tuples and passes the individual values to the function call. It also assigns the return value to the ReturnType object. A specialization for void returns is used to omit the return assignment part...

template<class R, class T, class F, class A, class B, class C> void Call(ReturnType<R>& out, Tuple<A,B,C>& t, T* obj, F fn) { out = ((*obj).*(fn))( t.a, t.b, t.c ); } 
template<class R, class T, class F, class A, class B>          void Call(ReturnType<R>& out, Tuple<A,B  >& t, T* obj, F fn) { out = ((*obj).*(fn))( t.a, t.b      ); } 
template<class R, class T, class F, class A>                   void Call(ReturnType<R>& out, Tuple& t, T* obj, F fn) { out = ((*obj).*(fn))( t.a           ); } 
template<class R, class T, class F>                            void Call(ReturnType<R>& out, Tuple<     >& t, T* obj, F fn) { out = ((*obj).*(fn))(               ); } 

template<         class T, class F, class A, class B, class C> void Call(ReturnType<void>&,  Tuple<A,B,C>& t, T* obj, F fn) {       ((*obj).*(fn))( t.a, t.b, t.c ); } 
template<         class T, class F, class A, class B>          void Call(ReturnType<void>&,  Tuple<A,B  >& t, T* obj, F fn) {       ((*obj).*(fn))( t.a, t.b      ); } 
template<         class T, class F, class A>                   void Call(ReturnType<void>&,  Tuple& t, T* obj, F fn) {       ((*obj).*(fn))( t.a           ); } 
template<         class T, class F>                            void Call(ReturnType<void>&,  Tuple<     >& t, T* obj, F fn) {       ((*obj).*(fn))(               ); } 

Thanks for the explanation.

I had figured most of it during the weekend.

The 'Get' functions are needed to convert the value "&T::a" into the type of that value, "F". N.B. this is a pre-C++11 trick.

I was having problems converting &T::a into a type, solved it by using decltype(&T::a) (C++11 ftw ).

ArgFromSig can then be passed a member-function-pointer type, and can extract the return type into a typedef, and the arg types into a typedef of a tuple.

template<class F> struct ArgFromSig;
template<class R, class T>                            struct ArgFromSig<R(T::*)(void )> { typedef ArgStore<R,Nil,Nil,Nil> Storage; typedef R Return; };
template<class R, class T, class A>                   struct ArgFromSig<R(T::*)(A    )> { typedef ArgStore<R,  A,Nil,Nil> Storage; typedef R Return; typedef A A; };
template<class R, class T, class A, class B>          struct ArgFromSig<R(T::*)(A,B  )> { typedef ArgStore<R,  A,  B,Nil> Storage; typedef R Return; typedef A A; typedef B B; };
template<class R, class T, class A, class B, class C> struct ArgFromSig<R(T::*)(A,B,C)> { typedef ArgStore<R,  A,  B,  C> Storage; typedef R Return; typedef A A; typedef B B; typedef C C; };

//ArgStore is just a wrapper around Tuple that adds some reflection capabilities:
const static uint MaxArgs = 3;
struct ArgHeader { void (*info)(uint& n, uint o[MaxArgs], uint s[MaxArgs]); };
template<class R, class A=Nil, class B=Nil, class C=Nil>
                                    struct ArgStore                { ArgHeader b; typedef Tuple<A, B, C> Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length; o[0]=offsetof(Type,a); o[1]=offsetof(Type,b); o[2]=offsetof(Type,c); s[0]=sizeof(A); s[1]=sizeof(B); s[2]=sizeof(C); } };
template<class R, class A, class B> struct ArgStore<R,  A,  B,Nil> { ArgHeader b; typedef Tuple<A, B>    Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length; o[0]=offsetof(Type,a); o[1]=offsetof(Type,b);                        s[0]=sizeof(A); s[1]=sizeof(B);                 } };     
template<class R, class A>          struct ArgStore<R,  A,Nil,Nil> { ArgHeader b; typedef Tuple       Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length; o[0]=offsetof(Type,a);                                               s[0]=sizeof(A);                                 } };                 
template<class R>                   struct ArgStore<R,Nil,Nil,Nil> { ArgHeader b; typedef Tuple<>        Type; Type t; static void Info(uint& n, uint o[MaxArgs], uint s[MaxArgs]) { n = Type::length;                                                                                                                      } };					   

Is there any reason to pass the return type from ArgFromSig to the ArgStore?

If you can use LuaJIT, you might find it easier to write C wrappers for all the class functions and invoke them through the FFI.

I'm already using LuaJIT so I'll look into it.

I wanted to understand how does the templates method work because it might be useful in the future :)

Is there any reason to pass the return type from ArgFromSig to the ArgStore?

Nope! Thanks for pointing that out That code just grew organically into it's current form, so apparently there's some useless leftovers in there.

I could've cut ArgStore out of the example completely, because it's reflection/Info feature isn't used.