Hello, I'm trying to learn more about building a multi server mmorpg environment. I already solved a couple of potential problems I could foresee but it seems obvious to me someone already solved these or had better design ideas. My problem is that I don't come from a networking background and what I learnt, I learnt by trial and error so I don't even know what I should read up on. It's a pretty classic problem I guess when getting serious about a new topic, I can't study some aspect of it or techniques if I don't know they exist. So I'm looking for some suggestions on things I could read about that would help building a multi server mmorpg. Just to give some background, I understand what's involded in running a single server mmorpg with a decent amount of users (decent on an in indie scale). I used to work for a small company running a mmorpg called Star Sonata before I moved on to EA. We were two engineers on the project for a almost a year and so I've touched most aspects of running a mmorpg, from database to networking. I'm trying to do this to get a more formal knowledge of networking and get some real experience in building a multi server mmorpg for learning purposes. Thanks!

Did you read the Forum FAQ?

Yes, I did actually just before posting this because I thought this might be a commong subject. I assume I did miss something so I re read it. I'm guessing you're thinking of "Q1) I want to start learning about networking. Where do I go?". I did read it the first time and all I saw was winsock tutorial but I hadn't actually gone to the multiplayer section of this site, I'll admit I thought it was just another winsock tutorial. The article " Cyberspace in the 21st Century: Part Five, Scalability With a Big 'S'" and "Distributed Gaming" are of interest indeed.

Most of the articles I read about scalability seems to suggest using a kind of "front end" server that acts as an interface that just relay data between the servers that run the real simulation and the client. I'm not sure I like that model, I'm looking into making a fairly "twitch" game for my tests and they all mention this incure a good penalty in terms of network latency. Of course, if it's just me testing I dont suppose I'll see much of a difference but I'd like to assume real world conditions. I suppose I'd have to do some tests since I lack real world experience with this model.

Then I also had the issue of handling data that need to be accessed by many servers very often. Most would need to keep a local copy of the object. This brings up a lot of potential problems about consistency, delay in changing the data/propagating the change and for me the biggest is consistency in case of a crash of one of the servers. There's many solutions, have a way to resolve which local copy is valid, make sure the central version is always up to date, just reboot the servers and have all the other servers go down and restart with the same backup as the server that just went down etc.

You would probably need to find some way to parallelize your game - e.g. Everquest "zones" - such that you can run the same process on many computers. Suppose you have each machine simulating a different portion of your world map. Then, they need a method for communicating to the other machines "Player X has left my map and enters yours, here are his details". Put your computers on a fast LAN (dual 100 megabit or gigabit links) and your latency internally will be far less than that experienced by the outside world. I think the one gateway to the Internet is unavoidable, though. You'd need to put something on your LAN that can talk to the outside world and to the computers inside, and then a way to pass packets to the correct machine for those coming in. There is probably some specialized hardware out there that can do this sort of specialized routing for you, and then you'd lease a phat pipe and not worry about bandwidth : ).

There are also distributed file systems (PVFS comes to mind) which are designed with clusters in mind and are supposed to be "safe". You might also want a very scalable database system to store information about items hosted on one machine - all others would simply make accesses to the DB. Since database actions are atomic, you can cut out a lot of work on avoiding concurrency issues. Plus database access is generally fast, and would suit your MMO needs well. Backup is a snap.

How big are you planning to make this thing?

Quote:Original post by Dark Rain
...Most of the articles I read about scalability seems to suggest using a kind of "front end" server that acts as an interface that just relay data between the servers that run the real simulation and the client. I'm not sure I like that model, I'm looking into making a fairly "twitch" game for my tests and they all mention this incure a good penalty in terms of network latency. Of course, if it's just me testing I dont suppose I'll see much of a difference but I'd like to assume real world conditions. I suppose I'd have to do some tests since I lack real world experience with this model...

I assume you're approaching this as a small independent developer, rather than as an EA project. There are two main problem with implementing a 'twitch' based game - firstly a very wide 'pipe' is required to your shard - you're better off implementing a client-to-zoneserver structure than a client-to-networkforwarder-to-zoneserver. Flatter is better.

This can be expensive. Your intra-server (zone-to-zone) communications and synchronisations have to be fast and efficient, or you have to have isolated zones (no intra-zone communication). To this end, hardware wise, you're effectively wanting a number of zoneserver machines, each with an external interface (ideally fast), all on one secure internal LAN. Additional database or worker (such as AI solvers) servers would be on the same LAN. Basically, you're looking at either a friendly colocation centre, or shelling out for the priviledge of setting up a secure infrastructure in a managed datacentre. You wouldn't even want to think about running this sort of thing behind a domestic broadband connection, even a good one. You're needing a datacentre with a very good backbone.

You'd also, depending on available hardware and bandwidth have a seriously reduced player capacity per zone. This isn't so bad for indie games - we don't need the massive support infrastructure a commercial game expecting tens of thousands of subscribers in its first day does. We don't *need* those tens of thousands of players to cope with our costs. We can launch on one box, and grow our games much more slowly.

Quote:Then I also had the issue of handling data that need to be accessed by many servers very often. Most would need to keep a local copy of the object. This brings up a lot of potential problems about consistency, delay in changing the data/propagating the change and for me the biggest is consistency in case of a crash of one of the servers. There's many solutions, have a way to resolve which local copy is valid, make sure the central version is always up to date, just reboot the servers and have all the other servers go down and restart with the same backup as the server that just went down etc.

What you're effectively talking about here is zone gamestate synchronisation. Having a twitch based game, the additional traffic of transferring very rapid events in a synchronised manner between zones requires a good cluster infrastructure (LAN). If you're allowing realtime activities to pass between zones you really need a very specific means of forwarding only appropriate events to appropriate zones. I have a couple of long meandering threads [linky] [linky] regarding the use of a 'relevance graph', effectively a hierarchical scene graph for event propogation and attenuation in a multiserver environment. You may find it useful.

Yes, it's not an EA project but it's still something I'd like to use in a presentation plus it's just fun to go back to mmorpg programming.

To be clearer, I was indeed thinking of going for medium to small zones connected together. My initial idea was to create zone servers that run a bunch of zones and connect directly to them. I was starting to question myself as all the articles I read in game programming gems or on gama sutra suggested this was not the right approach.

I'd like to go for a vehicle combat game which is kind of similar to what I was working on before and having such a type of twitch game isn't all that bad in itself as far as required bandwith goes. The biggest trouble we had, was getting the server to run it's update cycle fast enough with a couple thousand of AI roaming the map, all the planets moving in near a thousand different galaxies etc so that the latency was low enough. The game was designed to be very dead reckoning friendly so that helped too.

I'll admit I hadn't given much thought to data center hosting as it's just a learning thing but who knows, it might catch the eye of a designer in the studio(I whish). When I was an indy just buying a box in a datacenter and adding a second one as we grew was easy. I'll admit I did wonder a lot what's most datacenter policy toward local traffic. Is it charged as regular bandwith, is it frowned upon etc.

Also, thanks for the word "zone gamestate synchronisation" it helps to have precise terms on concepts =). I'll also read the threads you suggested. My idea was having objects being registered as observers of a certain state, which in turns translate to servers being registered as observers of a certain state. I thouhgt I could have different sorts of state observed for each type of object so that the amount of data sent is limited to what's needed only.

I wanted the data to be shared mostly for aggro purposes. Did I just fire on a team causing it to be aggro to all my possesion, say a few harvester I have running in another zone on another server? I have a couple of idea to really limit this kind of traffic by accepting some imprecision of a few seconds maybe.

Thanks!

It's not simply zone borders. That would be much easier in a way. I have to consider that the player can own objects that affect the gameworld in completly different area of the simulation but these objects need to constantly have info about the player since say his skill in engineering or what not might give them more range or if he just aggroed a team of killer rabbit and there's a couple of killer rabbits on the same team near one of your base, they need to know they can attack those.

That's a much more precise question than your initial question :-)

There are multiple solutions:

1) Re-design the game to not need this support.

2) Build a distributed simulation infrastructure where objects can subscribe to the kinds of data they need, and changes are distributed as needed.

3) Slice simulation into different domains, where some domain could contain the player data and all objects the player owns within that domain; the domain then communicates to the world WRT state changes.

We've done both 2) and 3).

Yes, I agree it's much more precise but that's because I was hoping there existed a kind of "must have read" book or what not on this topic. Most domains I've studied have such work but I guess it *is* a pretty narrow and new field of game programming and networking.

As for my question, it was just an example of the first problem I came upon. My solution was the first practical solution that came to my mind after reflecting on this for a bit and so I imagined there would be someone who devised a clever way of solving some of the issues this has. Not that I don't trust my ideas, I just know that quite often someone studied the subject much more in depth than me and so they have a very good chance of having come up with a better solution.

Anyway, thanks for helping me confirm that my solution was correct, at least for this problem =).