Sign in to follow this  
Camille

MMO server architecture choice

Recommended Posts

Hi, I'm currently working on a mmo game (mmorpg to be precise). For the server part, I can see 3 solutions, but without knowing the one is the best/easier. a/ 1 server, 1 world This is the simplest solution at the beginning. Do a simple server, and when it's full, buy a new one. You'll then have 2 worlds. They're full ? Buy a third one, and so one. Pros: - easy to develop - easy to grow Cons: - multiplication of game communities - adding functionnalities (then ressources need) to a server means lowering maximum players count. b/ 1 world, n specialized servers Here, we split the server logic in different servers. Once for the connection, another one for IA, the third for fights, one for chat, ... All of these servers need to be connected one with the other. Pros : - architecture 'easy' to understand - everyone on the same world, without bigger servers - code well partitionned. each server has his own utility, and don't need to konw how his friend works. - good for team work. A server for everyone, everyone a server :p Cons : - not very scalable : what happens if the chat server is not used, but if the fight one is full ? - probably needs a lot of communication between servers, for data (characters states, ...) exchange - synchronisation problems ? c/ 1 world, n parallel servers In this solution, instead of specializing each server, we just do copies of the same. For example, we could imagine a simple server do all the work (fights, chat, ...) but only for a portion of the world. When a player cross an imaginary boundary, he just connects on another server, and that's all. Pros : - scalable. Just add a new server, assign some parts of the world to it, and the load will be balanced. Cons : - need to split the world in smaller entities - probably needs a lot of communication between servers, for data (characters states, ...) exchange - synchronisation problems ? Well, here I'm in my reflexions. At this time, the last solution is the one I prefer, but I would like some opinions. Is there some traps I didn't see ? Does another solution I didn't see exist (maybe a mix between b and c) ? Thanks in advance for your help, and sorry for my crap english, Camille

Share this post


Link to post
Share on other sites
Out of all the MMOs I've played, they all seem to favor a 1:1 mapping of server-to-world. You may end up with multiple communities; but, that can add to the flavour of the game. You get some servers saying that they're better than others, and that could draw some people to those particular servers. One big question is though, would you allow each subscriber to create at least one character on each server, or would you limit them to one character per account?

It could also boil down to the workload your server needs to perform. If you don't have a player cap on a server, and everyone flocks to that one, you could experience severe lag, and possible server crashes. If you decide to stick to a single world, where every player plays on, then the parallel servers approach may be something to consider.

This also brings up the issue of costs for moderators/game masters. If you were to use the 1:1 mapping, then you would need to find GMs for each world.

As well, some players prefer to do nothing but PvP while playing an MMO. Others prefer PvE. So you should take that into account. If you decide on keeping with a single world, you could alienate one of those two groups if a balance isn't found. Personally, I'd prefer a dedicated PvP server, yet still have the option of becoming a PvP player on the regular servers. But, if I were to become a PvPer on a regular server, I don't think I should be allowed to liberally attack non-PvP players, unless they either consented to a sort of duel, or entered a PvP "arena".

Just some food for thought.

Share this post


Link to post
Share on other sites
Hi there. It's nice to see another MMO project kicking off from someone who's actually considered the back end architecture. ++ for you.

For an indie with limited funds, I think option c is the closest to what should be done, but remember that a 'server' is not a machine- it's a process, which may be on a particular machine. If you keep the concept of process and machine separate in your mind your task becomes easier.

I've not heard of fights being handled on separate servers - 'heavy duty' AI tasks (A*) do tend to be farmed off to an AI process, but this relies on the AI process having the requisite resources loaded for all tasks.

Personally, the architecture I advocate is a microcluster consisting of a single master process, a login process, and a database process, not necessarily on the same machine. Optionally, slave processes can take over a portion of the game world, by registering with the master process. There's a fair few threads on this sort of thing in this board.


Share this post


Link to post
Share on other sites
Thanks for your wuick answers.
Quote:
Original post by Dorvo
It could also boil down to the workload your server needs to perform. If you don't have a player cap on a server, and everyone flocks to that one, you could experience severe lag, and possible server crashes.

It's what's happening with the last game I worked on. It's was a simple architecture, a mix of the a and the b solution (one server for conn, one for database, and the last one for... everything else). Connection server and database process are fine. But balancing game servers is a trap. At this time I've got the oldest game server holding about 4.000 simultaneous players when the 2 last ones take only 2.000 of them. Add to this GM problems, as you said, and you know why I'm trying to find a different way :p
But of course, you're right, the best solution is still to have several different worlds. At least to separate PvP, PvE and Roleplay.
i'm not against a "p worlds, n x p servers (or process)" solution :)

Let's continue with a new question :
What about servers emplacement ? (Ok, it's no more development here, and I would understand if someone complains about it)
For lag and performance, it's always better to have the server and the players as near as possible. That means you have to set up different servers, at least one per continent. So my question : do you think it's better to think about it as several different games, with no connections, or to think it as just some parallel worlds, linked not with a LAN but with a WAN ?

Ok, I stop my philosophical questions, now :p

Share this post


Link to post
Share on other sites
Typically the login process is not time-critical and so can be centralised (physcially, and also tied in with your subscription management). Individual clusters (world instances) which an authenticated client can communicate with can then be localised (per continent or whatever). The player simply sees a single game, which they choose a shard of, and know that they can expect better performance in a local shard.

Share this post


Link to post
Share on other sites
We do 1 world, N "interchangeable" server machines. This is great for a "virtual world" type environment. The fact that the servers are interchangeable (scale linearly) means that you get pretty easy redundancy/fail-over, too. We also do seamless hand-off between the servers, so you never see any zoning.

However, this took a while to get right. I would not recommend it for someone who's starting an indie project. Perhaps even more importantly, there are significant game design aspects that are hard about a single world, and hard about a non-zoned world.

Zones provide structure to a game world, and I've heard several game designers say that, after being zone-less, they'll go back to zones. Just like platformer games still have levels -- you need that structure.

Further, if you design a single world for 200 people online, what will you do when you have 1000 people? If you design it for 5000 people online, what will you do when you have 25000? Shards solve this (very hard) game design problem; they allow you to balance a single world to a specific population count, and then replicate it as necessary while maintaining balance.

If I were to build a MMO today, with a heavy RPG game-play element, I would go for a zoned world (one process per zone), and scale through sharding. It's absolutely simplest technically, and solves a number of game design problems.

Share this post


Link to post
Share on other sites
Thank you for this concise and concrete answer !
It confirms my opinion to go with zoned servers and to add shards when required by the gameplay (size of the world too small).

Btw, I'm not alone on this project, and already built up a 15.000+ simultaneous players mmorpg. So the seamless zoning system seems to be an attracting challenge, now :p

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
[QUOTE]
Typically the login process is not time-critical and so can be centralised
[/QUOTE]

Keep in mind that there are times that your login server can be time-critical. The biggest is when the game crashes, and around 6:00 PST. You will get a flood of clients trying to log on.

I've worked on a game with 30,000+ simultaneous users. When the game crashes, you get 30,000 login requests in the span of a few minutes. If you can't elegantly deal with load like this in your login server, then you end up with a huge problem, as people keep connecting/disconnecting to try to get online.

Just something to think about. The login server may not do much of the work most of the time, but there are those critical time it NEEDS to be responsive under load. I know firsthand how aggravating, and negative an unavailable login server can make players.

Share this post


Link to post
Share on other sites
Quote:
Keep in mind that there are times that your login server can be time-critical. The biggest is when the game crashes, and around 6:00 PST. You will get a flood of clients trying to log on.


Why? Your authentication ticket should still be valid; the client doesn't need to re-authenticate unless you're down for longer than your ticket expiry time. Or do you generate a new random server secret (and re-distribute it) each time (some part of) a server goes down?

Share this post


Link to post
Share on other sites
Given a very large player base, I'd probably split the login processing across several login servers - say one per cluster. I'm assuming that 30k on a single cluster is unlikely (more like 5-6k concurrent) - and assuming that players can select which cluster they wish to play on before connecting. This allows us to distribute the load somewhat, with clients connecting to one of a range of login servers, rather than trying to get through one box.

Alternatively, if a single cluster architecture is used for the entire game (or a very large player base per cluster is intended), then a simiple solution to handling many concurrent login requests is to use mirrored login databases, with connections being made to the appropriate login process by crude load balancing (first server refuses connection, move to second and so on), before passing the authenticated connection to the cluster master process, which will assign it to the appropriate zone process.

In any case, processing time for login details is not as critical an issue as in-game response times.









Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by hplus0603
Quote:
Keep in mind that there are times that your login server can be time-critical. The biggest is when the game crashes, and around 6:00 PST. You will get a flood of clients trying to log on.


Why? Your authentication ticket should still be valid; the client doesn't need to re-authenticate unless you're down for longer than your ticket expiry time. Or do you generate a new random server secret (and re-distribute it) each time (some part of) a server goes down?


I keep my tickets valid only long enough to do the handoff to the player's communication server. (My model may vary from others. I don't connect directly to gameworlds, but have an intermediary communications server that data is routed through).

Depending on whether or not the server that has just gone down is critical, determines if the system does an emergency shutdown, or if it can recover.
Having redundant nodes is helpful, and in many cases will avoid a catastrophe, but in instances when a bug has surfaced that compromises data integrity, there is no choice but to shutdown the entire game system.

Most of my experience is not in MMORPG, however, it's in console networking. So there may be differences.

Many problems we had were not from the game servers crashing, but from upstream routers crashing, or flapping due to load, or ISP issues. Having multihomed servers on different backbones helps, but when one of the routers on your backbone goes out, a large chunk of your players drop, and they race to login again. This happens also after maintenance, or other various times.



Share this post


Link to post
Share on other sites
Guest Anonymous Poster
There is a mixed architecture for all of this:
-one login server
-I database servers (to load blalance the system)
-J zone servers (with cross server chat)
-and optionally K update servers

Where I,J,K is any number that works for the game.
Where there are more zones than zone servers, the
server processes can share one physical server.

Viktor

Share this post


Link to post
Share on other sites
Quote:
I keep my tickets valid only long enough to do the handoff to the player's communication server.


I would recommend, if you want to avoid having a login system slammed when lots of people come back, that you issue tickets that are valid for longer times (say, four hours at a time?) and re-issue them every two hours while the player is connected. That would let a player re-connect to any of your servers directly (without going through separate authentication) as long as they re-connect within two hours.

Btw: for other readers: by "ticket" I mean something that contains the player ID, ticket expiration time, other identifying data if needed, and a hash (Tiger, MD-5) of these items plus a cluster secret. This allows you to get the player ID out of the ticket and verify that it's valid without running anything more expensive than a hash, when the user connects.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by hplus0603
I would recommend, if you want to avoid having a login system slammed when lots of people come back, that you issue tickets that are valid for longer times


Yea, It's a tug-of-war sometimes trying to decide what to do. I really like the idea of the player having to manually verify his password on connect. Having a ticket lying around in the system makes me nervous, as there could be possible security implications.

Do I want the possibility of someone taking the ticket that is on my machine, and use it to login without any verification? Do I want my little brother to be able to log on to the game as me, without a password.

As long as he gets on with the ticket, he can play indefinately as the ticket will renew every 2 hours. Granted, I can do some sort of client side authentication, by comparing a hashed password against one that is in the ticket. But that still doesn't stop people stealing the delicious cookie and using it for evil.

I really like the idea of the ticket being a server hashed portion of a larger whole. A whole that can only be completed with a second peice of data that has been hashed with the user's password...

How about this:

The server knows the player's password.

The player is authenticated with his password, and then sent a ticket that has been signed by the server, and encrypted with the user's password. In order for the player to be handed over to the game server, the player must take the encrypted ticket, unencrypt it with his password, and send the original hashed ticket back to the server. The server verifies the consistency of the ticket, and that it has been properly signed.

The server then immediately issues the player a new ticket (processed in the same manner), the player does not decrypt it when he recieves it, but lets it sit there.

The server continues to issue new encrypted tickets as long as the player is active.

If the game crashes, or the player gets knocked off, the player is left with an encrypted ticket. He is asked for his password when logging back on, but does not go to the login server, instead he decrypts his ticket with the password, and sends it off to the game server.

This way we still maintain that a player know his password to get back online, but do not bother with the login server. In fact, the login server now becomes more of a load balancer, to determine which server the player needs to connect to.

I think I would be happy with this kind of scenario..





Share this post


Link to post
Share on other sites
There is one complete mmo open-source client and server system out there which has been used for a commercial game (ryzom).
Engine (set of libraries) is called Nel (www.nevrax.org).
Take a look at it, you can learn a lot and have a good network layer.
Its a huge code base, but thats the deal with mmo's.

I used it a couple of years ago for a mmo demo I coded, including writing a set of services etc.
If my memory does not fail, I remeber that it used 1 world n servers.
Or rather.. you have one or more "front-ends" (request routers) and customnameserver.
Then you have x numbers of "services", which are exe's providing a specific service and handles a specific set of commands. For example a positioning service or a chat service.
These service can be distributed over one or several machine.

So a service only has to know the main gateway/custom nameserver, then it tells it that it's alive and who it is and the gateway broadcasts this etc. And later the router knows where to route these kinds of calls.

It's in general a good filosfy, since you can start of with a couple of sevrers and easaly grow by separating out the most loaded/intensive services to one or more separate machines.
I ran it successfully with about 10 services distributed over two machines + a separate machine runnign an mysql server and php/apache front for login etc.

I think it also had software fail-over mechnism (in the router/gateway/nameserver) etc.

Well.. it was a while ago.. check it out.

/martin

Share this post


Link to post
Share on other sites
The ticket lifetime really only matters if you disconnect unexpectedly. For a regular "logout" you could send a message to the server that you're logging out, which would invalidate the ticket on the server. This requires a "ticket id" as part of the ticket, of course. You could also include things like the source IP address in the ticket, to avoid spoofing from other machines.

If you're worried about a technically savvy little brother (who knows enough to bypass the client-side login screen), then I wouldn't write the ticket to disk; just keep it in memory while the client is still running. That way, you can re-connect to the servers at will while you're still "logged in" in the client process, but once you quit, the ticket is gone. (Unless someone can fish it back from the page file ;-)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this