So I've done 2d and 3d single player games and have been a professional game developer for awhile now but I haven't done any serious networking programming. My goal is to write a basic/working mmo server architecture, it may never be complete however I want to give it a shot. I've already decided to go the route of C++ and enet but that is the only thing I'm solid on at the moment. I took some time to design out what I think would be a decent server architecture but I'd like some input on my design. My Server Design I haven't really found detailed resources on this subject although I'm sure it's out there some where, if you know a design that is better I'll love to see the link. [Edited by - Seoushi on April 29, 2009 10:57:09 PM]

I don't know how MMO servers are setup on the backend to handle changing zones and all that stuff, but up until that point, your design looks fine. It actually resembles a MMO I am familiar, Silkroad Online, almost perfectly. I do know that game does not use a separate chat server though, but aside from that, I've seen that design in practice, so nice work!

I've also seen some games that split the login server into two servers. One is the account server, which is your typical login process and the second server is a character server, where that server handles choosing a character to login with or creating/deleting, etc...

I've also seen some games that do it all on one server, but your design is way more scalable and flexible. However, you do have more points of failure. I've seen some games that use the 2/3 server layout get hit hard when one specific server goes down and interrupts the service. Proper server coding helps alleviate such problems, but it is something to be aware of. Some might see that as a good thing as well, since if the login server has a problem, then people in game won't have any issues until they relogin, so it just depends on how you feel about that.

Quote:I haven't really found detailed resources on this subject although I'm sure it's out there some where, if you know a design that is better I'll love to see the link.

I don't think there are many resources and especially not many detailed resources simply because of the compromise that could ensue if all games started blueprinting their designs [wink] That's one of the best benefits of knowing assembly and how to reverse engineer, as you can put together your own resources on what's being done on the front end as well as the game networking design.

I would be interested in hearing from anyone on how things are done on the backend especially with the whole master world server and slave world servers that handle different zones and whatnot. I still can't get a clear understanding of how to go about designing for that.

Here's a couple of threads that might interest you:
Distributed server architecture for load balancing
Collection of MMO related posts (Self plug of my own thread, but 20 good threads referenced you might want to read for sure!)
MMO Mapserver Boundaries

You probably don't need N login servers. One (or two, if you don't want to have a single point of failure) is enough. People don't want to log in 25,000 times per second, and it takes a while to load a game anyway, so even if a login takes 5 seconds, that's fine.

I would let the clients connect to worlds through the "gateway" servers, not directly. This will take away considerable load, and it will make some exploits harder.
For example, the typical "drop and crash the server" type of dup exploits don't work so well if you don't connect to the server, nor know its address at all (ok, these should not work anyway because you hopefully use ACID transactions... but you know, nobody is perfect, you can never be sure that you didn't leave an exploit open somewhere). Now, if the client connects via the gateway, then a forced crash will have the only effect of a few connected users being kicked out, it won't affect the world state, nor the players on the neighbouring nodes.

Patch servers are best implemented as standard web servers (apache, thttpd, or whatever) that serve signed update files. Have the client pick one by random, don't worry which one. If it fails, use the next in the list. This won't give every server perfect minimal load, but it will be close.
That way, it's ridiculously cheap and easy to set up and administer, there are no security troubles, and it will work just as good as a much more complicated solution. You even take advantage of proxy servers, reducing your bandwidth needs. The client can tell genuine patches from the signature, and there is no problem if someone "pirates" patch data -- let them, it's public data anyway.

Looks pretty reasonable.

The master/area server is definitely a common pattern.

I don't know if a dedicated "chat" server is necessary. YMMV.

I'd consider having the client connect to the "gateway" server, and have it just forward messages to the appropriate world servers, rather than making new connections. That ends up being a little easier.

As far as threading, avoid it. Use non-blocking IO and just single-thread your processes. You'll save yourself a lot of headache. And since you'll probably run multiple processes per physical box, you'll still be taking advantage of multiple cores.

Don't get too crazy with the load-balancing to start with. It probably won't be necessary. Content, rather than CPU, is often the limiting factor, so it's likely that instancing will be a better solution to load-balancing. Dynamic load-balancing sounds neat, but in a lot of cases it's better to just tell individual processes what areas they'll own and go from there.

On a micro-level, the more you build your architecture around messaging, even in a single proc, the easier you'll find things go. Don't ever block on anything.

A remarkably useful thing is to have a separate logging server. Really, it sounds stupid, but is actually pretty useful.

Quote:Original post by _Kami_

i disagree about using master server as proxy to avoid new connections. it will be a major bottleneck and single point of failure.

I've worked on (large) systems that both used a central connection point and didn't. The ones that used a central point ended up being much easier to deal with than the ones that didn't. If load is a problem (it's highly doubtful that it would be, unless your serializatioin requires silly amounts of CPU), spawn multiples of them and round-robin 'em.

You give a startup sequence for the order the servers should be started in. To me, this sounds like a bad idea; if your gateway server goes down, you want to be able to bring it back up without stopping and re-starting you world servers.

You should be able to add other methods of registration, for example a gateway server could send out a request for all the other servers to re-register with it.

Thanks for all the replies, I didn't figure this thread would get much attention.

The startup sequence is a good point. I think I will keep it the way it is for setting up but if the gateway fails it should broadcast itself. Another way would be to have the gateway broadcast every now and then for new servers. Perhaps there should be multiple gateway servers, 3-5 at max and their basic role is to take over in case one fails.

As for no threading, I could do that however I see the servers being more scalable to multi-core computers if you can just tell it how many worker threads to handle. Doing threading doesn't look bad with boost::threads and especially since I only have two places for shared data locks and mutexes won't be too much of an issue.

The reasoning I have behind the separate chat server is so you don't have to have the game loaded to chat with people in game. For all I care it could be an irc server. It also makes it easier to communicate between servers. Chat could also include voice as well as text so having separate chat servers makes sense in that instance.

From what I've read login and patch servers can get hit pretty hard and I've seen it myself (timeouts on login). Having multiple login servers is a must imho. As for patch servers being http/ftp servers that was pretty much my idea, they don't need to be very complex and existing technology such as apache will work just fine.

Benton, thanks for the links at first I was thinking of doing all movement calculations on the servers however after reading one of those threads I realize even most commerical mmo's don't do it. Now my plan for that is just having sanity checks every now and then.

As a side note I'm thinking of making the servers somewhat like a botnet. For example a server will start up with the generic server then if load gets too high it will ask for a certain type of server (login, world, master world, gateway) and that server will become it. As load becomes less of an issue servers can be demoted to their original botnet state. This would save energy and make it easier to scale the world.

I believe that addresses most of the concerns expressed so far. I realize that this design is quite complex and if I was to make a game based on it that I wouldn't have to have nearly this much scalability (since the population most likely wouldn't get above a couple thousand players) but since I'm doing this for fun I don't see the harm in making it too scalable :) .


EDIT:

I've appended to the end of the original page a revised version of my idea incorporating some suggestions from this thread and some things I've heard about through research ( Here ).

[Edited by - Seoushi on April 29, 2009 10:09:54 PM]

Quote:Original post by Seoushi

As for no threading, I could do that however I see the servers being more scalable to multi-core computers if you can just tell it how many worker threads to handle. Doing threading doesn't look bad with boost::threads and especially since I only have two places for shared data locks and mutexes won't be too much of an issue.

To take advantage of multicore systems, you can also use multiple processes per machine.

Quote:As a side note I'm thinking of making the servers somewhat like a botnet. For example a server will start up with the generic server then if load gets too high it will ask for a certain type of server (login, world, master world, gateway) and that server will become it. As load becomes less of an issue servers can be demoted to their original botnet state. This would save energy and make it easier to scale the world.

I think this will be more complexity than you really need.

Quote:I believe that addresses most of the concerns expressed so far. I realize that this design is quite complex and if I was to make a game based on it that I wouldn't have to have nearly this much scalability (since the population most likely wouldn't get above a couple thousand players) but since I'm doing this for fun I don't see the harm in making it too scalable :) .

The more time you spend working on your scalability, the less you have to work on your game. In all things, balance.

This is your friend:

send(destination, message);

If this is used for most communications, even internal ones, then moving stuff between servers gets a LOT easier. Look up "Actor model" on wikipedia.

Quote:The more time you spend working on your scalability, the less you have to work on your game. In all things, balance.

True, however I don't really have a game in mind yet. The basic plan is to get the architecture in place then start adding in basic game functionality such as moving. The end result will hopefully be generic enough for most mmo's.