[Lua] Multi threading Lua...

ZealGamedev · 2010-10-21T15:10:24

I have been seeking help on the official Lua forums, but not many people post there, so I thought I would try my luck here. To recap my problem -I have two C threads, and two lua_States (created via lua_newthread(mMainLuaState)). I have a function setup that ensures correct data parallelism, so in theory thread1 and thread2 should be able to both call the function at the same time (the two calls have their own environment, and only read static read only globals). However, I am getting one crash after another (no helpful information provided in the logs sorry).Check out my original thread for more details and example code...http://forum.luahub.com/index.php?topic=2596.0There is no way I am the first person to try this... somebody out there must know what I am 'missing'?*fyi I have NOT defined lua_lock/unlock, but for what I am trying to do, I should not have to right?*here is the example code to save you from having to check out the other thread...Lua-- Create a unique message buffer for each threadmsgs2 = {}msgs3 = {}-- Declare two unique functions for each threadfunction foo2() table.insert(msgs2, 777); endfunction foo3() table.insert(msgs3, 777); endC mBarrier = new boost::barrier(3); // The main lua state is initialize (mLua), create two 'sub states' mLua2 = lua_newthread(mLua); assert(lua_type(mLua, -1) == LUA_TTHREAD); int lua2Ref = luaL_ref(mLua, LUA_REGISTRYINDEX); assert(lua_type(mLua, -1) != LUA_TTHREAD); mLua3 = lua_newthread(mLua); assert(lua_type(mLua, -1) == LUA_TTHREAD); int lua3Ref = luaL_ref(mLua, LUA_REGISTRYINDEX); assert(lua_type(mLua, -1) != LUA_TTHREAD); // Spawn two threads which use mLua2/mLua3 to call foo2/foo3 boost::thread thread2(boost::bind(&Sim::lua2Task, this)); boost::thread thread3(boost::bind(&Sim::lua3Task, this)); mBarrier->wait();void call(lua_State* lua, std::string& func){ lua_getglobal(lua, func.c_str()); assert(lua_type(lua, -1) == LUA_TFUNCTION); if (lua_pcall(lua, 0, 0, 0)) Aux::log(lua_tostring(lua, -1));}void Sim::lua2Task(void){ for (int i = 0; i < 10000; ++i) call(mLua2, std::string("foo2")); mBarrier->wait();}void Sim::lua3Task(void){ for (int i = 0; i < 10000; ++i) call(mLua3, std::string("foo3")); mBarrier->wait();}

Engines and Middleware Programming

Started by ZealGamedev October 20, 2010 04:33 PM

24 comments, last by ZealGamedev 13 years, 6 months ago

ZealGamedev

100

Author

October 21, 2010 01:55 PM

Quote:Tables are garbage collected, they aren't allocated on the stack

Really? I always assumed all local allocations avoided the GC and just got deallocated when they went out of scope. Maybe some special cases are needed for closures/upvalues? But thats not the case in my example...

So it sounds like the GC is the primary reason why there is a problem (although you mentioned there could be other problems, like hooks ect...)?

y2kiah

2,032

October 21, 2010 02:18 PM

Quote:Original post by ZealGamedev
Quote:Tables are garbage collected, they aren't allocated on the stack

Really? I always assumed all local allocations avoided the GC and just got deallocated when they went out of scope. Maybe some special cases are needed for closures/upvalues? But thats not the case in my example...

So it sounds like the GC is the primary reason why there is a problem (although you mentioned there could be other problems, like hooks ect...)?

I think the moral of the story here is that without intimate knowledge of Lua's inner workings you should do what the developer recommends. Ask yourself if it's really worth your time to fight it, or should you instead accept what it is and work within those constraints. Maybe you've determined that it is worth your time, I don't know...

http://jeffkiah.com

ZealGamedev

100

Author

October 21, 2010 02:28 PM

Quote:
I think the moral of the story here is that without intimate knowledge of Lua's inner workings you should do what the developer recommends. Ask yourself if it's really worth your time to fight it, or should you instead accept what it is and work within those constraints. Maybe you've determined that it is worth your time, I don't know...

Oh I agree. While I am 100% confident this problem could be solved (for starters, a thread safe GC like Sneftel suggested), I am also 100% confident it would require a lot of work. Like I said, I am just surprised this issue has not been addressed before (maybe it has?).

I suppose moving all my data into c and routing all communication through there is looking more and more like the best option. Perhaps the benefits (as described by the bitsquid guys) will out weigh the costs.. Still a shame though.. if only the lua GC/threads behaved slightly differently...

rip-off

11,000

October 21, 2010 02:50 PM

Quote:Original post by ZealGamedev
Like I said, I am just surprised this issue has not been addressed before (maybe it has?).

It was a design trade-off. There is typically no easy answer to such questions.

Making each lua_State thread safe would be very expensive in a single-threaded system. Additionally, the benefits might not be great because such locking inhibits parallelism, and such locks are likely to be highly contended.

Threading is platform dependant, Lua is AFAIK written in pure C with little or no platform specific code.

blog | twitter

y2kiah

2,032

October 21, 2010 03:09 PM

Quote:Original post by ZealGamedev
I suppose moving all my data into c and routing all communication through there is looking more and more like the best option. Perhaps the benefits (as described by the bitsquid guys) will out weigh the costs.. Still a shame though.. if only the lua GC/threads behaved slightly differently...

well not all your data, maybe just data that needs to be thread safe. Personally I solved the problem by exposing my event framework (which is inherently thread safe) to Lua so I can trigger events from Lua and listen for/handle events in Lua. If I need to send some data to another thread, I package it in an event and set up a listener in the other thread. I find that working through a framework is a solid way to implement threading rather than worrying about and solving the same problems in many places.

http://jeffkiah.com

ZealGamedev

100

Author

October 21, 2010 03:10 PM

Quote:Making each lua_State thread safe would be very expensive in a single-threaded system. Additionally, the benefits might not be great because such locking inhibits parallelism, and such locks are likely to be highly contended.

It wouldnt necessarily have to be all that expensive. I am sure there are some clever lock free designs that would work. And it could always be a optional feature you choose when building (kinda like the current optional lua_lock/unlock defines). However, for now I think we are going to just try the two state solution and migrate all our data into C. Perhaps in the future we will revisit this topic (assuming the two state solution doesnt work out).

Anyway, thanks everyone for the input. I had just assumed that a lua_newthread() state would be independent of the main state/GC at LEAST for local allocations, but now I know.

[Lua] Multi threading Lua...

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

[Lua] Multi threading Lua...

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines