Sign in to follow this  
the_edd

Seeking scripting language with preemptive threads

Recommended Posts

Hi folks. I'm trying to find a suitable scripting language for my application. The part that is scripted will generate a DAG of tasks. Child tasks must complete before parent tasks can be started. Think of it like the graph constructed/implied by a makefile. I'm thinking that these tasks will be constructed by some kind of closure mechanism, so all I have to do is "call" a task to start it running. There are some obvious ways to parallelize the execution of the tasks once the graph is constructed, but I'm struggling to find a nice scripting language with good support for threads. After some quick tests it appears that Python's GIL will bite me hard. I then looked at Lua and I can't see any way that I can wedge (preemptive) threads in there. I was wondering if I could pull the constructed tasks out of a lua_state and execute them inside another state in a different thread? This would be my ideal solution as Lua is a good fit otherwise. Stackless Python looks quite interesting, but I suspect it will be a bit more of a pain in terms of distribution. Are there any others that I should know about?

Share this post


Link to post
Share on other sites
Stackless Python gives you cooperative (non-preemptive) threading, not preemptive threads. Stackless's threading model is pretty much the same thing as Lua coroutines.

Are you sure you need preemptive threading? The most significant difference between cooperative and preemptive threading, is that you can do symmetric multiprocessing (as in, utilizing 2 or more processor cores) with preemptive threading, and you can't do this with just cooperative threading. But I'm not sure if that's a requirement for you. If not, I'd really recommend cooperative threading, since it's much easier to write code for.

I don't think it's good to share a lua_state between threads, but there are other things you can do. Check out this thread for ideas on doing real multithreading with Lua. It looks like it's possible, but as expected there's a lot of things to be careful about.

Share this post


Link to post
Share on other sites
Without pre-emptive threading, you don't need semaphores, mutexes, or other access-control mechanisms; you can rest assured the code executing right now is the only code flow in the middle of a function, except suspended coroutines. Generally, this is very important in a scripting language.

I've never seen a scripting language using preemptive threading models, but I suppose you could create a host-language (C, C++, Java, etc) light user data, fill it with methods that allow (synchronous, thread-controlled) access to a data store, and then put it into the globals table of multiple Lua states, and run each state in a separate host-space thread.

Share this post


Link to post
Share on other sites
Quote:
Original post by pinacolada
Stackless Python gives you cooperative (non-preemptive) threading, not preemptive threads. Stackless's threading model is pretty much the same thing as Lua coroutines.


Ah, a pity. Thanks for pointing that out.

Quote:
Are you sure you need preemptive threading?

I don't absolutely *need* it but it's highly desirable. If you look again at the problem, you'll see there is a massive amount of exploitable parallelism inherent in the problem, and the kinds of tasks being run will be quite long-running.

Quote:
If that's not a problem, I'd really recommend cooperative threading, since it's much easier to write code for.


I'm quite happy writing code using pre-emptive threads. I just need a scripting language with good support for it. I could do this in C++, but then it wouldn't be as extensible as I would like.

Quote:
I don't think it's good to share a lua_state between threads, but there are other things you can do.

I wasn't thinking of sharing it as such. I was thinking of somehow copying part of it in to a new state and handing that state off to another C++ thread. This may not be possible, though.

Quote:
Check out this thread for ideas on doing real multithreading with Lua.


After meandering around the thread you posted, I came across Pluto, which might help me move Lua "stuff" between states. Thanks for posting the link!

Share this post


Link to post
Share on other sites
Quote:
After some quick tests it appears that Python's GIL will bite me hard. I then looked at Lua and I can't see any way that I can wedge (preemptive) threads in there. I was wondering if I could pull the constructed tasks out of a lua_state and execute them inside another state in a different thread? This would be my ideal solution as Lua is a good fit otherwise. Stackless Python looks quite interesting, but I suspect it will be a bit more of a pain in terms of distribution



Cooperative multi-tasking is generally about single-threaded execution. This is also the greatest advantage it offers, since it removes the need for protecting shared state (in the way threads do).

If you need to run it on multiple cores, you run one interpreter on each, and pass data in between in same way you would with threads (applies to Python as well).

Cooperative multi-tasking is simply minimal overhead multi-tasking emulation. For true concurrency, you still work in concurrent world, and all the existing problems exist.

Task-based parallelism generally isn't a matter of language, it's a high-level concept, usually a part of library.

Quote:
I'm quite happy writing code using pre-emptive threads. I just need a scripting language with good support for it. I could do this in C++, but then it wouldn't be as extensible as I would like.


Run X interpreters, each in its own thread, pass tasks to execute via sockets (gives you even networked concurrency), or by some lockless FIFO.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this