So I've been binding some C++ functions to Lua and all has been going well until I came across this problem.

local transform
transform = gfx3D.Transform(); -- Works fine (transform is a userdata)
transform.Position = Vector3(0, 0, 0) -- This seems to work fine (Position is a userdata)
transform.Position.z = -80 -- This doesn't work (z accessed with a __newindex call, position with an __index call)

LOG(tostring(transform.Position)) -- Prints X:0 Y:0 Z:0

Both transform and position are userdata but setting z does not work. I wondered why this was until I realised exactly the reason. This is the __index method for the transform userdata:

static int TransformGet(lua_State* L)
{
	if (luaL_checkudata(L, 1, "Transform") == NULL)
	{
		luaL_typerror(L, 1, "Transform");
	}
	Transform* trans = (Transform*)lua_touserdata(L, 1);

	const char* key = luaL_checkstring(L, 2);

	if (strcmp(key, "Position") == 0)
	{
		Maths::Vec3f* returnValue;
		returnValue = (Maths::Vec3f*)lua_newuserdata(L, sizeof(Maths::Vec3f)); // Creates a new vector for the return value, and so breaking the reference
		luaL_getmetatable(L, "Vector3");
		lua_setmetatable(L, -2);
		*returnValue = trans->Position;
	}
	else if (strcmp(key, "Rotation") == 0)
	{
		Maths::Vec3f* returnValue;
		returnValue = (Maths::Vec3f*)lua_newuserdata(L, sizeof(Maths::Vec3f));
		luaL_getmetatable(L, "Vector3");
		lua_setmetatable(L, -2);
		*returnValue = trans->Rotation;
	}
	else if (strcmp(key, "Scale") == 0)
	{
		Maths::Vec3f* returnValue;
		returnValue = (Maths::Vec3f*)lua_newuserdata(L, sizeof(Maths::Vec3f));
		luaL_getmetatable(L, "Vector3");
		lua_setmetatable(L, -2);
		*returnValue = trans->Scale;
	}
	return 1;
}

As you can see when I access the Position element in the transform I create a new userdata and push it on the stack as the return value. But this breaks the reference to the original userdata, and so setting it's Z element has no effect on the original, and so it appears the assignment didn't work.

The logical answer is to return a reference to the existing data of the position vector. But it doesn't have it's own userdata as it's simply an element of the transform userdata. You can see the transform userdata being created here:

Transform* transform;
transform = (Transform*)lua_newuserdata(L, sizeof(Transform));
luaL_getmetatable(L, "Transform");
lua_setmetatable(L, -2);
*transform = Transform();

I am completely at a loss at what to do here. I somehow need to create a userdata that points to the same block of memory that the position element was kept in. I tried experimenting with lightuserdata as it's only a pointer, but that threw errors when I tried to access members of the lightuserdata, which seem to not exist in Lua's mind.

This seems like an issue other Lua devs must have come across so I'd like some help on dealing with it.

Thanks.

I tried something like this:

static int TransformGet(lua_State* L)
{
	if (luaL_checkudata(L, 1, "Transform") == NULL)
	{
		luaL_typerror(L, 1, "Transform");
	}
	Transform* trans = (Transform*)lua_touserdata(L, 1);

	const char* key = luaL_checkstring(L, 2);

	if (strcmp(key, "Position") == 0)
	{
		lua_pushlightuserdata(L, (void*)&trans->Position);
		luaL_getmetatable(L, "Vector3");
		lua_setmetatable(L, -2);
	}
        
        return 1;
}

But sadly it did not work. I get this error from Lua:

LuaSource/main.lua:26: bad argument #1 to '__newindex' (Vector3 expected, got userdata)

Only tables and full user data can have per instance metatables. All other types have per type metatable, I think only string type metatable is set to the string global table/library by default.

In my opinion you are overengineering here. Lua type system with metatables, raw get/set and few others, is very screwy - mapping C++ classes 1:1 to it is a disaster waiting to happen. I'd just make a metatable for full userdata Transform and then give it setters and getters or have them merged into one function and depending on number of arguments pick if it's get or set for everything: positionX that can take double (sets position.x) or nothing (returns position.x) and so on for each component of your transform. If you need Vector3 as userdata too, then add a function position() that returns the vector userdata instance that is the copy of the vector or takes one at sets it's value to transform (or takes 3 numbers, to avoid cost of constructing userdata if you want to set all 3 components at once).

There is literally no reason to have 'nice' and 'C++ like' API in Lua IMO, it's a tiny performance hit (not noticeable but still), it's harder to write both C(++) and Lua code for that, it's harder to reason about. Too much downsides.

Lua has almost no limit on this kind of things because of metatables and everything, but it doesn't mean you should do it.

It's hardly overengineering to want to be able to do this is it?

transform.position.x = 10

Is there really no way for the position getter to return a reference to the same data?

You could push full userdata that has in it a pointer to that Vector3 in Transform and has type/metamethods for field access do assignment/reads through that pointer, but that makes the new type incompatible with real Vector3 full userdata, and it means that when your Transform gets garbage collected, the Vector3ref instance that points to the vector will seg fault. Unless you do some reference counting or something... light userdata is just pointers and one pertype metatable and I usually use them only for registry entries and for upvalues in C functions, because that way I'm 100% sure what is what.

Maybe there is some easy way to do this with returning tables that have metatable with methods that have upvalues set to this vector... I'm not sure and it'll have the problems of keeping track of what is what (real Vector3 vs. Vector3 'ref' to one in Transform) and what if GC takes Transform while we reference the Vector3, it's.. just not worth it to me.

Most straightforward way to me in pure Lua is to just do positionX(), positionY(), position() and so on functions and make them 'overload' on number of arguments in their implementation.

If you use C structs only, then Lua JIT has powerful mechanism to make access to these very fast and easy and it parses C struct declarations, but there are few caveats there too (metatable is per struct type, permanent and unchangeable once set, you tie yourself to LuaJIT, there is additional list of caveats and conversions related to use of C structs in Lua ect.)

I can see the need to simplify and maybe change my approach to things. It's a little disappointing, but I may have gone into lua with the wrong mindset.