I've been thinking lately about ways to serialize game state, and I began thinking about how this would work in a game using a script interface that supports cooperative threads. (Coroutines, or generators, or what have you.) Let's take a simple example and suppose we have a map with an object moving around it whose motion is controlled by a coroutine. Then say we want to save the game at some point, and be able to resume it later with the object picking up exactly where it left off. It seems to me that one option is to build a system in which cooperative threads can be serialized while they are suspended, and then revived at a later time with the same locals and resume point. The game could have a scheduler that manages all cooperative threads, and can serialize them to and from persistent memory along with all the other data that makes up the game's current state. In this way, the behavior of objects within the game can be scripted with cooperative threads without worrying about what happens if a save-and-reload occurs in the middle of their lifetimes; as long as the all the objects are brought back in the correct state, and the scheduler restores itself to the same state as before, things will work smoothly. Another option would be to just serialize your entire VM state, though that seems like an unnecessarily heavy operation. My question is, is this a feasible solution in general? Does anybody actually do this, or is it generally better to restructure your game so that all cooperative threads will terminate before game state is saved? (Or else make script writers operate under the assumption that their threads are not part of the game's persistent state.) I did a bit of searching, and found this thread, which suggests that Lua is capable of doing what I describe, but I'm still curious as to whether this seems like a good idea or not. It seems like it would have a few obvious benefits, including making it possible to write simple, natural entity scripts while basically hiding most serialization from the script interface. It would involve a bit more work on the back end, but I'm not seeing any significant reasons that this wouldn't be a worthwhile option. Any thoughts?

Yes, it's entirely possible (and quite easy, actually) to do all this with Stackless Python.

The only real drawback, as far as I'm concerned, is performance. If you have some kind of simulation with thousands of agents, creating a microthread for each one is not a practical approach. But if you're only dealing with hundreds at a time, it's not an issue.

Quote:Original post by KulSeran
Tell that to eve online. They use it because it is the most performance friendly way for them to manage all the thousands of agents in their game.

No, it really isn't the most efficient. I posted benchmarks on this forum a while ago showing the overhead of running Stackless tasklets compared to plain function calls. I expect Project EVE, like any MMOG, is able to divide the workload between many servers.

Quote:Original post by drakostar
Yes, it's entirely possible (and quite easy, actually) to do all this with Stackless Python.

Do you have a reference for how to go about serialising each of the tasklet states?

Quote:Original post by drakostar
No, it really isn't the most efficient. I posted benchmarks on this forum a while ago showing the overhead of running Stackless tasklets compared to plain function calls.

And do you have a link for this? I'm sure that cooperative threading in Python isn't ever going to be the most efficient way to do anything in performance terms, but it would be interesting to see how much you gain in simplicity compared to a more efficient approach. EVE appears to scale far better than other MMOGs so Stackless is doing something right.

I researched this a little bit - since the whole cooperative thread thing feels like such a powerful programming metaphor - individual entities that are in fact sequential programs. Being able to save them off as pure state would be a really nice bonus.

These are links that I found related to the ocaml programming language. I can't really comment with very much insight unfortunately, since I am still only learning ocaml (I spend most of my time looking at the ffi and trying to work out the best way to map c++ types with ocaml types instead of profitably exploring ocaml).

There is a library extension that supports serializable delimited-continuations in ocaml. The delimiters operate as a check on the depth of the bytecode-stack frame that gets copied. Unfortunately it only works with ocaml bytecode and not native code (you would have to make use of native OS support for creating multiple os-based stack frame objects in order to support a true native code implementation). However being a statically typed language, you could probably expect that the ocaml bytecode will run faster than stackless python. I got as far as compiling and installing it - but have never really played with it -

links :-
continuation framework on top of which you could build coroutines

using serialization (to disk) of the continuations in a cgi application

Perhaps more interesting - since it is more lightweight, and implemented purely using ocaml language features , is lwt . It is a cooperative thread library built upon a monadic framework. I suspect that you might be able to use ocaml marshalling and channels with this, since ocaml marshalling supports serilization of closures and functions i believe. I've only installed it - and a basic contrived example works fine, using either the native or byte code compilers. I dont really have a good grasp on monads however and so cant pursue it as I would like to, or comment informatively.

I think i read that someone did make a serializing scheme for lua and coroutines - but it was an individual effort that didnt attract much interest. I've lost the link now i am afraid.

More generally, I would be woried about state that is bound up in the os layer (and which therefore cannot be easily be captured) and serialized since it doesnt percolate up to the language level. I am thinking things like network connection state etc. I suspect that these types of considerations which would need to be factored out of the serialization scheme would weigh heavily and dictate a lot of the application design.

Disclaimer - I am a bit new to all this - so if it sounds like i got something wrong - then I probably did!!
[edit]for clarity hopefully

Quote:Original post by Kylotan

Quote:Original post by drakostar
Yes, it's entirely possible (and quite easy, actually) to do all this with Stackless Python.

Do you have a reference for how to go about serialising each of the tasklet states?

Unless I'm misunderstanding the question, this should cover everything:

Quote:One of the main features of Stackless is its ability to pickle and unpickle tasklets. That means that a running program inside a tasklet can be persistently stored to a file or string. Later, it can be restored again and can continue to run at the point where it was previously halted.

Quote:

Quote:Original post by drakostar
No, it really isn't the most efficient. I posted benchmarks on this forum a while ago showing the overhead of running Stackless tasklets compared to plain function calls.

And do you have a link for this?

Unfortunately I can't find it now (I'm *sure* I posted it on this forum), but my recollection is that not counting initialization time, running a series of 1000 or so tasklets (which just generated a few random numbers to waste CPU cycles) took 25% longer than calling a function 1000 times. Feel free to write your own benchmark and see. A more realistic test case would probably reduce that difference somewhat.

Quote:I'm sure that cooperative threading in Python isn't ever going to be the most efficient way to do anything in performance terms, but it would be interesting to see how much you gain in simplicity compared to a more efficient approach. EVE appears to scale far better than other MMOGs so Stackless is doing something right.

Oh sure, Stackless would let you do really cool things like serialize an AI-controlled object and pass it off to another server with no interruption. I just happen to be in exactly the wrong situation where I can't use it, which is the only reason I complain about performance: developing a single-player game with a huge real-time simulation that should run decently on older hardware.

Woops, it was a different forum. Here is my test code.

try using stackless python

Quote:Original post by drakostar

Quote:Original post by KulSeran
Tell that to eve online. They use it because it is the most performance friendly way for them to manage all the thousands of agents in their game.

No, it really isn't the most efficient. I posted benchmarks on this forum a while ago showing the overhead of running Stackless tasklets compared to plain function calls. I expect Project EVE, like any MMOG, is able to divide the workload between many servers.

Why would you compare tasklets to pure function calls? The first lets you do things that the second doesn't. A good performance comparison would be tasklets versus some other threading implementation.