Sign in to follow this  
PaePae

Python choices for MMO?

Recommended Posts

I have no experience with Python before. But was in a team developing one local commercial online game and one prototype, both were in C++. I am now interested in Python. After research around I found that I have to make decision with choices... Python: CPython / Stackless Python / PyPy Network: Standard Lib / Twisted / (more choices?) CPython is main stream and has more library directly supported. Stackless is well-known for its concurrency. PyPy is also included concurrency feature from Stackless. - Stackless seems to attract me most, but I do not know if there is any network engine supported, or I should develop my own? - Or choosing CPython with a network engine is more suitable? - PyPy seems not mature to me. Can I rely on PyPy for production use? - Python 2.6 or 3k ? I found much less library supporting 3.x :-( I can develop my own network engine, but existing and reliable engine/library is preferred so I can concentrate on game content. I am also interested in RakNet, but existing RakNet-Python binding is outdated, sadly. I would like to hear your opinion and suggestion. Thank you :-)

Share this post


Link to post
Share on other sites
I have a bit of experience with Stackless Python - the fellow responsible for it, Christian Tismer, is a coworker of mine. Stackless Python is an excellent choice for MMO development. If you've heard of EVE Online, that was written entirely in Stackless.

I think you'll find it to have very satisfactory networking support.

As for PyPy, I can't really offer any comment. I don't have enough substantial experience with it.

Good luck with your project!

Share this post


Link to post
Share on other sites
Quote:
Original post by cg123
I have a bit of experience with Stackless Python - the fellow responsible for it, Christian Tismer, is a coworker of mine. Stackless Python is an excellent choice for MMO development. If you've heard of EVE Online, that was written entirely in Stackless.

Not entirely in stackless. Components of it are in stackless though. Others are in C.

The networking support of stackless is the same as that of Python, which is good enough for a reasonable sized game. You will find advantages though in moving to a more ASIO (IOCP for instance) setup, as EvE did not too long ago.

Share this post


Link to post
Share on other sites
From what I understand, Stackless Python isn't actually "stackless" per se, but instead it moves parts of what used to be on the C stack onto an internal stack, and then it uses fibers for the C parts.

What I don't understand is whether Stackless gets rid of the global Python interpreter lock, or whether that's still a scalability problem? Would Stackless not have an internal data structure problem if two threads tried to update the same dict at the same time?

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
From what I understand, Stackless Python isn't actually "stackless" per se, but instead it moves parts of what used to be on the C stack onto an internal stack, and then it uses fibers for the C parts.

What I don't understand is whether Stackless gets rid of the global Python interpreter lock, or whether that's still a scalability problem? Would Stackless not have an internal data structure problem if two threads tried to update the same dict at the same time?

The GIL is still in place. While that doesn't stop you from having your C parts multi-threaded, it does mean the python parts wont be. There have been movements to refactor the GIL out of python in general, those have been resisted by morons who hate the idea of doing more than one thing at once. Refactoring the GIL out "is" possible, just not easy.

Yes, stackless uses fibers (on windows, something else on linux) to implement its stacklessness

Share this post


Link to post
Share on other sites
The problem with the GIL is that it severely restricts in-process scalability across cores. Even five years ago, that wasn't much of a problem: "pizza box" servers were cheap and single threaded, and you would simply scale across processes instead. However, these days, I would not choose infrastructure that does not let me take advantage of multiple cores at the same time. This is my biggest beef with the modern Python ecosystem in general.

Thus, I would disagree with the recommendation that stackless is great for an MMO design that you start today. Note that EVE was started something like 8(?) years ago. In the future, you'll have to do more than what EVE is doing to be competetive, and they're already doing all they can with the current Python infrastructure.

Share this post


Link to post
Share on other sites
Yes and no. You simply run multiple processes, or take advantage of ASIO/IOCP. Mind you, EvE is not entirely python, a large chunk of it is C/C++ and takes advantage of threading there, the python core is more used for tasklets, aka microprocesses.

Eve's been running fine with only minor hickups along the way, and their only real major issues have been the lack of IOCP usage, and certain traffic problems from their non-deterministic combat. That's with 40k people on the same cluster simultaneousnesly. Python will scale if you work with it. Their biggest slowdowns actually came from the number of transactions they were running on the SQL cluster, and on a lot of the heavy mathematics required for certain portions of the game. After switching to their new IOCP architecture and running ramsan's things are very responsive even in some of the most heavily loaded systems. Plus they can allocate an entire server to just a single cluster if need be in order to handle high processor issues (such as when two 500+ fleets engage in battle). This is something no other MMO has ever done (when was the last time you saw an MMO with 1+k people engaging in combat, along with between 5 (standard) - 20 (motherships/carriers) drones per person, or in other words, around 6k-21k elements requiring tracking, calculations, and feedback being passed around in realtime?

Share this post


Link to post
Share on other sites
Here's my question on the whole issue about using Python for a MMO or even a MMO emulated server (more in my interests atm):

Can we, the average network programming Joes and Janes, actually come up with such a system that resembles (a smaller scale of) EVE's on our own by simply using the language itself or is there still a lot of special domain knowledge required?

For example, it is my understanding that in Erlang, concurrency, distributed processing, and fault tolerance mechanisms are an inherent part of the language in such a way that you just code your stuff and handle the specific events of when something goes wrong and everything is taken care of you due to how the language itself works. I know that's an oversimplification of it all, but just follow.

Whereas, in C++ when using Winsock, you have to specifically code and design an architecture that supports thread safe operations for concurrency, develop your own protocol for a distributed system, and work in your own external means for handling fault tolerance and process restarting (as hplus0603 gave a good example of what they did here. I know it's two different things being discussed, but just a loose example.)

So, do you actually need to know what you are doing before you do it when you use Stackless Python as the tool to implement a MMO, so that if you don't know what you are doing (yet), there's no real benefits to actually using it? Or, is it something that will allow you to come up with a solution that might be "good enough" on an Indie scale and you can generally just improve upon it as needed if you wanted to reach EVE levels?

I ask because every time I see "Stackless Python", I see EVE mentioned, but nothing else. I would be inclined to think the former of my previous question and that people who are trying to learn the ins and outs of it all aren't going to benefit from using it because we simply don't have the experience required for a successful solution. Any thoughts on this?

Share this post


Link to post
Share on other sites
Quote:
Original post by Drew_Benton
Here's my question on the whole issue about using Python for a MMO or even a MMO emulated server (more in my interests atm):

Can we, the average network programming Joes and Janes, actually come up with such a system that resembles (a smaller scale of) EVE's on our own by simply using the language itself or is there still a lot of special domain knowledge required?

No matter what, domain knowledge will be required. No platform, and certainly no language, will remove that requirement.
Quote:
For example, it is my understanding that in Erlang, concurrency, distributed processing, and fault tolerance mechanisms are an inherent part of the language in such a way that you just code your stuff and handle the specific events of when something goes wrong and everything is taken care of you due to how the language itself works. I know that's an oversimplification of it all, but just follow.
If this were only true. Except its not. That gross oversimplification you mentioned... it's important. Those details are what ultimately make life interesting/difficult. It's not as simple as "I just write my code and it magically is concurrent and distributed!"
Quote:
Whereas, in C++ when using Winsock, you have to specifically code and design an architecture that supports thread safe operations for concurrency, develop your own protocol for a distributed system, and work in your own external means for handling fault tolerance and process restarting (as hplus0603 gave a good example of what they did here. I know it's two different things being discussed, but just a loose example.)
Actually, the two end up being about the same in complexity probably. Writing an asynchronous erlang network server and a C++ one will encounter the same kinds of issues, and will end up having to solve them.
Quote:
So, do you actually need to know what you are doing before you do it when you use Stackless Python as the tool to implement a MMO, so that if you don't know what you are doing (yet), there's no real benefits to actually using it? Or, is it something that will allow you to come up with a solution that might be "good enough" on an Indie scale and you can generally just improve upon it as needed if you wanted to reach EVE levels?

You're not going to magically be able to just go mmo = MakeMMO() and expect it to work. Even if there was some magical uber architecture that took care of that for you, you would still need to know a great deal about what you're doing. Stackless is just a tool, its a means of dealing with micro-processes, which can be used to simulate many things in a game world. EvE isn't all stackless, and large chunks of it are in C/C++ and do work OUTSIDE of the stackless simulation. But stackless is an important part of EvE because its ultimately what runs the game simulation.
Quote:
I ask because every time I see "Stackless Python", I see EVE mentioned, but nothing else. I would be inclined to think the former of my previous question and that people who are trying to learn the ins and outs of it all aren't going to benefit from using it because we simply don't have the experience required for a successful solution. Any thoughts on this?

How many MMOs are out there now? How many of them are actually like EvE? I'll bet most MMOs use scripting on the backend in some form or another, just because it's the only way to reliably be able to change the game mechanics on the backend trivially. We just don't hear about it much.

EvE's use of python is brought up frequently in this arena simply because they've put a lot of technical knowledge out there on their use of python and how they have benefited from it (rapid prototyping, ease of use, and more). Most MMOs are very secretive about their architectures, they don't want you to know how they work on the backend. EvE practically has documentation on their site as to how the whole thing is put together.

[Edited by - Washu on September 13, 2009 1:48:51 PM]

Share this post


Link to post
Share on other sites
For Drew_Benton's question, that was also what I asked myself.

After I did research and tried weighing many options, as Washu already stated, the main point I am willing to use Stackless or PyPy is its concurrency mechanism. Also many documentations and presentations that EVE developers provided letting us to think out of our old technical models and implementations.

I am not going to build a giant realtime world as of EVE, but just some casual turn-based online games. So performance is not an issue yet, especially when EVE showed us they can greatly dealing with. GIL is not an issue too, my recent projects use multi-processes model and they work fine. So what light I see in [Stackless] Python is the nature of scripting language and the Actor model that will ease many issues I found in old days with C++. And yes, Python is capable with C/C++ extension as needed :-)


About GIL, since Python 2.6 they added multiprocessing package to help migrating thread-based to process-based concurrency and it can utilize multi-core CPU. However, be aware that serializing and communication between processes is expensive.



My still question, is there any network engine recommended for building an MMO?

Share this post


Link to post
Share on other sites
Quote:
Actually, the two end up being about the same in complexity probably. Writing an asynchronous erlang network server and a C++ one will encounter the same kinds of issues, and will end up having to solve them.
I agree that domain knowledge is essential when dealing with complex issues.

However, I very strongly disagree with your claim of equivalent complexity.

Erlang networking lets you work at a level of abstraction where most things that are issues (and complicated ones) in other languages have already been solved for you.

I assume that by asynchronous network server you mean one that uses nonblocking io (hence no need for thread-per-connection). Here's a network server in erlang with the following characteristics:

1) No thread-per-connection.

2) Uses advanced polling if environment supports it instead of plain select() (epoll on linux, kqueue on BSD, don't know if erlang can use windows IOCP).

3) Work done by each connection handler will scale to multiple cores (this is a consequence of the way the erlang scheduler works on multicore systems now) but the whole thing will only use MAX_CORES system threads to run.

The equivalent of the above is very tricky to write in C++, as it forces you to more or less directly deal with issues 2 and 3. Not in erlang; this server responds to "hi" messages and echoes messages it can't understand:


-module(async_server).
-compile(export_all).

-define(TCP_OPTIONS,[binary, {packet, 4}, {active, false}, {reuseaddr, true}]).
-define(TIMEOUT, 500).

init(Port) ->
{ok, Listen} = gen_tcp:listen(Port, ?TCP_OPTIONS),
loop(Listen).

loop(Listen) ->
{ok, Socket} = gen_tcp:accept(Listen),
spawn_link(?MODULE, handle_conn, [Socket]),
loop(Listen).

handle_conn(Socket) ->
case gen_tcp:recv(Socket, 0, ?TIMEOUT) of
{ok, <<"hi">>} ->
%do some work
gen_tcp:send(Socket, <<"hi urself">>);
{ok, Received} ->
%do some other work
gen_tcp:send(Socket, <<"can't understand ", Received/bytes>>);
{error, Reason} ->
error_logger:error_msg("Socket error ~p", [Reason])
end.


It's pretty interesting that this snippet of code basically solves this nasty old problem out of the box, and is pretty trivial to extend to limit number of users, traffic-per-user, and other traffic shaping.

So I'd have to say choice of language for network programming can be extremely significant, at least to me; there are many projects that I'd write in erlang in an afternoon and that I wouldn't even want to attempt in C or python (generally those where performance is important, as that's the requirement that makes C and python networking ugly).

You can get an approximation of that ease, without the multicore scaling, in stackless python if you're willing to use the socket event wrapper contrib modules (stacklesssocket.py and socketlibevent.py) but I've been less than satisfied with them, ymmv (apart from the fact that they're not part of core stackless for some reason).

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous P

The equivalent of the above is very tricky to write in C++, as it forces you to more or less directly deal with issues 2 and 3. Not in erlang; this server responds to "hi" messages and echoes messages it can't understand:


I beg to differ.

While C++ is somewhat more verbose as language, developing this type of networking model is no longer hard.

C# comes with similar model as well.

Erlang has strong points, but this particular example is not necessarily the defining advantage. It also depends on type of networking and exact performance characteristics. And obviously, C++ will always lose due to verbosity, language nonsense, build process, ....

Share this post


Link to post
Share on other sites
Quote:
Original post by PaePae
I have a question regarding Erlang.

Can I just use Erlang as a networking and server platform, and write everything else in Python or C++ running on top of Erlang servers?


That's kind of the idea I had in the back of my mind when I made this post, Do you think this is a feasible server architecture design? (honestly can't believe I had a typo in the title and missed it), except I based my thinking along doing it in C++ simply because I know the language well enough to relate any answers I got about the idea. I personally don't know much about erlang other than the really simple programs I've tried, so trying not to bite off too much to chew at once, I just stuck with IOCP/C++.

I think the problem you will run into will be, which was pointed out to me more or less, the networking layer that handles the data processing isn't the problem that needs to be solved. I kind of see that point now after doing some more thinking about it. Let's say for a second you do have amazing network data processing servers that handles all that stuff, what about your game's logic servers and all those interactions?

If anything, you need your entire backend system to run on that platform (Erlang, in this case) to be able to really benefit from the language features. So, the problem as explanation to me is not of "How do I handle a lot of connections and data" as much as "How do you design your game to run in a distributed manner and not have the entire thing self-destruct?"

That's basically what I'm trying to figure out with the Python/MMO option because right now I am already looking into using Lua specifically for the benefit of being able to rapidly prototype out a MMO faster than I ever could in a non-scripting language. In my case, I can and want to sacrifice any performance possible as long as I am able to prototype out the design and really get an idea of "how things work".

For me at least, it's not about easy or hard or how much time something takes as much as just, I have no idea how things work, I want to learn and figure them out first hand. I have ideas of how things work, but it seems to be turning out they aren't so accurate. So, I'm trying anything I can think of and get advice on to build up the domain knowledge required to actually successfully implement such a solution. I've not read up on EVE's information they have released about their development process yet though, but that's certainly on the todo list.

The thing I was trying to figure out with my last post was, by using Stackless Python, will I actually be able to work through prototyping out a MMO using the language features it provides, or do I just not know enough yet to where there's no real benefit to me using it over any other scripting language for the task?

Of course, the answer to that in my case is that I should just try it myself after learning the Python language first, since trying to implement such a project in a language you do not fully understand will come around and bite you, as Washu so nicely put it. [wink]

Share this post


Link to post
Share on other sites
Quote:
Python will scale if you work with it


Much of Google runs on Python, so I think we can all agree with this :-)

Regarding the question: Does it help to use the platform, or do you need to know what you're doing? Sadly, you need to know what you're doing. The platforms are just various ways of organizing what you're doing, not ways of doing things per se.

Btw: the Forum FAQ has a list of MMO and other networking middleware packages.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
Regarding the question: Does it help to use the platform, or do you need to know what you're doing? Sadly, you need to know what you're doing. The platforms are just various ways of organizing what you're doing, not ways of doing things per se.


The platform to me says: We understand the problem, and we solved it, so you don't have to. Or, more commonly - the cost of our platform is lower than cost of you learning and developing from scratch.

Some vendors deliver on that promise. Many more don't, and rely on quite realistic chance that users will never push the limits, or even come close to them.

Too frequently, this accidental success leads to hype following that becomes self-fulfilling prophecy, until only years later users realize other options are better.

And here the catch-22 occurs. To understand which vendor actually delivers, one must understand the problem.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603From what I understand, Stackless Python isn't actually "stackless" per se, but instead it moves parts of what used to be on the C stack onto an internal stack, and then it uses fibers for the C parts.


This is so wrong I barely understand it. Stackless Python does not use fibers. It just manipulates the stack. But yes, it is not "stackless", not any more at least.

Quote:
What I don't understand is whether Stackless gets rid of the global Python interpreter lock, or whether that's still a scalability problem? Would Stackless not have an internal data structure problem if two threads tried to update the same dict at the same time?


Stackless as a fork of Python does not aim to change it, merely to extend it. Primarily with green threading, but also with the coroutine equivalent (can block the whole call stack, C frames included) channel communication mechanism and last but not least the ability to persist running code via pickling.

As such, there are no wacky changes like removing the GIL. It should be 100% compatible with existing code for standard Python, this includes both Python and C extensions.

Share this post


Link to post
Share on other sites
Quote:
Original post by WashuThe networking support of stackless is the same as that of Python, which is good enough for a reasonable sized game. You will find advantages though in moving to a more ASIO (IOCP for instance) setup, as EvE did not too long ago.


Yes, non-blocking IO is essential if anyone is going to make best use of Stackless. The "stacklesssocket" module I wrote can be googled, or if IOCP (on Windows) is desired, I've written the main part of an IOCP version at this Google hosted project.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603Thus, I would disagree with the recommendation that stackless is great for an MMO design that you start today. Note that EVE was started something like 8(?) years ago. In the future, you'll have to do more than what EVE is doing to be competetive, and they're already doing all they can with the current Python infrastructure.


They are?

I'd point out Second Life. They've been quite forthcoming with details about how they've implemented their scripting language and achieved scalability. That is, the ability to migrate running code from one node to another.

This is something that is possible with Stackless as well, and CCP are not using it at this time.

Share this post


Link to post
Share on other sites
Quote:
Original post by Drew_Benton
Here's my question on the whole issue about using Python for a MMO or even a MMO emulated server (more in my interests atm):

Can we, the average network programming Joes and Janes, actually come up with such a system that resembles (a smaller scale of) EVE's on our own by simply using the language itself or is there still a lot of special domain knowledge required?

For example, it is my understanding that in Erlang, concurrency, distributed processing, and fault tolerance mechanisms are an inherent part of the language in such a way that you just code your stuff and handle the specific events of when something goes wrong and everything is taken care of you due to how the language itself works. I know that's an oversimplification of it all, but just follow.

Whereas, in C++ when using Winsock, you have to specifically code and design an architecture that supports thread safe operations for concurrency, develop your own protocol for a distributed system, and work in your own external means for handling fault tolerance and process restarting (as hplus0603 gave a good example of what they did here. I know it's two different things being discussed, but just a loose example.)

So, do you actually need to know what you are doing before you do it when you use Stackless Python as the tool to implement a MMO, so that if you don't know what you are doing (yet), there's no real benefits to actually using it?


The real benefit that Stackless brings is the way it allows you to write more readable Python code. Got some logic you want to run? Launch it as a tasklet. Got something that looks remotely like a callback? Wrap it with a channel and have whatever invoked the logic that takes the callback look like a normal synchronous call.

Given you know about IO and that for the scheduler not to be blocked, non-blocking IO needs to be used (wrapped as described by channels) then you're pretty much set once that's taken care of.

You may have some idea that you'd like to use the tasklet pickling at some stage to build your own scalability solution, but that's something you can bring in in a way that best fits when the time is right.

I would say the answer is no.

Quote:
Or, is it something that will allow you to come up with a solution that might be "good enough" on an Indie scale and you can generally just improve upon it as needed if you wanted to reach EVE levels?


This to me reads as a description of the development of EVE. Of course, Stackless is so much simpler these days being a stable product. Back when EVE was initially being developed Stackless had just come out, was actually "stackless", was then based on continuations and went through rewrites.

Quote:
I ask because every time I see "Stackless Python", I see EVE mentioned, but nothing else. I would be inclined to think the former of my previous question and that people who are trying to learn the ins and outs of it all aren't going to benefit from using it because we simply don't have the experience required for a successful solution. Any thoughts on this?


There are other companies and projects out there using Stackless Python, however not all of them are interesting in discussing or divulging it.

Personally, I think people stick to the technologies that they know, trust and believe in. If they've bought into .NET, then they're going to find it hard to abandon it and jump over to something like Stackless. Same with C++ and other languages. I know there's one licensable MMO engine where the released EVE architecture papers were used to implement some of the systems without Stackless.

Share this post


Link to post
Share on other sites
Quote:
They've been quite forthcoming with details about how they've implemented their scripting language and achieved scalability.


Last I checked, Second Life claimed to support no more than 40 players per island (which maps to a server process).

Second Life are trying to compete directly with the OLIVE platform from Forterra Systems (where I work), so I won't say out loud what I think of Second Life claims about scalability, but I encourage you to do research based on all publicly available information, and make up your own mind.

I will note that, at the IETF forum on virtual world interoperability, several hard-core internet protocol architects expressed significant concern about the implementation of Second Life protocols with regards to correctness, security and scaling.

If you want to learn how to scale systems, I'd suggest you look at some systems that scale a lot higher than Second Life per instance, such as World of Warcraft, EVE Online, or perhaps upcoming games like MAG (256 players per "island"). Second Life largely scales by adding new instances (islands), which doesn't solve the problem of "more people want to meet in the same place."

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
Quote:
They've been quite forthcoming with details about how they've implemented their scripting language and achieved scalability.


Last I checked, Second Life claimed to support no more than 40 players per island (which maps to a server process).

Second Life are trying to compete directly with the OLIVE platform from Forterra Systems (where I work), so I won't say out loud what I think of Second Life claims about scalability, but I encourage you to do research based on all publicly available information, and make up your own mind.

I will note that, at the IETF forum on virtual world interoperability, several hard-core internet protocol architects expressed significant concern about the implementation of Second Life protocols with regards to correctness, security and scaling.

If you want to learn how to scale systems, I'd suggest you look at some systems that scale a lot higher than Second Life per instance, such as World of Warcraft, EVE Online, or perhaps upcoming games like MAG (256 players per "island"). Second Life largely scales by adding new instances (islands), which doesn't solve the problem of "more people want to meet in the same place."


You're responding to a selective sentence in my post, not the point I was trying to make. I do not care and have no interest in how well Second Life scales.

I was responding to your point about how CCP have used all that Python has to offer them and pointing out one functionality Second Life have implemented which CCP also has available through Stackless Python and do not use.

Share this post


Link to post
Share on other sites
Nobody has commented on Twisted, so I suppose I will: don't bother. It looks like a sexy framework until you try to actually do things with it. Like, say, use SSL. Oops, the SSL handshake is a blocking operation, which *blocks everything*. A small hiccup in one client will result in the server freezing forever. Fiddling with threads is no help. Twisted doesn't even have multiprocessing support.

The fundamentals of Twisted, the very basic things it should be able to do half-decently, seem horrendously broken to me. You can't do nonblocking SSL? You don't use multiprocessing, which is absolutely critical for Python scalability thanks to the GIL? What? I wound up writing my own stuff using nothing but the excellent standard library, which as of 2.6 features more or less full OpenSSL support. It's just a little testing server to aid my client development, but it ended up with fewer lines of code and far better functionality than the version I had written with Twisted.

Honestly, in a few hours you can probably recreate the bits of Twisted you actually need (using little more than asyncore) and do it right.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this