Scaleable game server system

Started by
3 comments, last by hplus0603 9 years, 8 months ago

Hi! I am currently designing multiplayer server system for our new game. I have some experience in building client-server systems but only for small amount of players. The game will be mobile game with multiplayer where players are playing against eachothers 1vs1 in real time. Direct connect from device to device is not possible because we want that players are able to play with random people around the world (for example clash of clans and hearthstone). The system should be scaleable so if player base grows up, we can simply launch more server nodes to handle them. I would appriacte if you guys could check this out and let me know if this can work. System will be run on AMS cloud computing servers.

Here are my plans so far:

Login server:

-Clients connect to this node when they launch the game

-Handles account login and redirect player to one of front-ends (balances the load between multiple front-ends)

Front-ends:

-Handles all network traffic between clients

-There can be multiple front-end nodes to balance network load

Lobby server:

-When player want to start multiplayer game they are placed to queue

-When another player with same matchmaking is found lobby server redirects the players to one of game server

Message server:

-Handles in-game messages from player to player

-If the players are connected to different front-ends this server will deliver the message to the right one

Global event server:

-There will be global cooldowns in game that should be updated even when player is offline

-Handle cooldown timers and for example notify players when cooldown or upgrade is finished

Database:

-Database cluster where all information is stored

Game servers:

-There are multiple game servers all connected to the front-ends

-One game server runs a networking thread and 1 to n game threads (thread count depends how many CPU cores are available)

-One game thread can handle couple of game sessions

So do you guys have some tips to make this better or are there any great bottle necks that should be fixed? Thanks!

gameServerSystem.png

Advertisement
Only things that stands out are the items that (I think) you've defined as single servers, e.g. login . I think you're under estimating the load they can come under. For instance, when the whole US eastern seaboard gets off work at 5pm and everybody starts logging in. You want to decentralize _everything_ that you can possibly can. Also allow spinning servers up and down dynamically, handle servers crashing as gracefully as you can, etc.

Also, what's your inter-server communication format? AMPQ server? Custom protocol? Are these designed to run in a data center together? A cloud server?

Sean Middleditch – Game Systems Engineer – Join my team!

So, in general, you have three kinds of services:

1) Batch services -- login, inventory, player stats, etc

2) Matchmaking services -- lobby, finding another player to play with, etc

3) Game services -- actual real-time gameplay data exchange while playing

All services in option 1) should probably be built as a horizontally scalable webapp. Use a round-robin load balancer in front of the web servers, and spin up as many web service instances as you need. Amazon has Elastic Load Balancer as a service. Or you could spin up a single instance or two of HAProxy or similar, and call that good (but beware the lack of redundancy!) Typically, all such services talk to the same database back-end, and that back end is horizontally sharded. Using in-RAM databases (like Redis,) and/or key/value stores (like Redis or Cassandra,) and/or separate caching (like memcache) may improve scalability. Just don't use MongoDB. Friends don't let friends use MongoDB :-)

Services in option 2 and 3 may need to be in the same instance, if you're going to be doing advanced things. Or you can simply have the matchmaker spin up a new process on some of a number of "game" servers, and identify those "game" servers with IP-address-plus-port, and give eash participating player an identification token and the ip+port information. Each of the two players then connect to the same ip-address-plus-port and provides their identifying token, and the game server can know that they go together. (In fact, you can have multiple player pairs in the same process -- you shouldn't need more than one process per core, and one port number per process.)

If you want large amounts of players (over tens of thousands) chatting/matchmaking at the same time, you will need a scalable solution for that. If all you're doing is chat and matchmaking for up to 10,000 players at the same time, a single-instance will be significantly simpler and cheaper; beware of single point of failure for this matchmaker, so you may want to have a second copy already running but idle, and be prepared to swap over to that if the first one dies.

And, if you're in Amazon, you may want to replicate all customer data between two different availability zones, and have all server roles running in two different availability zones, to avoid the problems that happen when one particular zone (or data center) dies, which seems to happen about once a year.

Finally, the matchmaker service, and the game services, can end up calling your web service back-end for things like verifying game tokens, granting players special things, etc.

If you build the system like this, you can start with very small amounts of server hardware (maybe even only in a single availability zone, if you're "small and indie.") Then you can add hardware instances as needed when and if you grow.
enum Bool { True, False, FileNotFound };

Having 10,000 people chatting in the same room is probably not what you want, text would be scrolling past so fast people would have chance of reading, so you can easily divide that player base into instances.

I think you are over-designing your architecture a bit. You might want to consider login, front-ends and lobby servers in to one or multiple servers runnin on the same physical machine with public ID.

Under-estminating the stress of a login service is something I seen other developers do, but then again I think it is just a matter of choosing the right tool for the job. For example my NextGen game server www.next-gen.cc handles about 10,000 logins in a few seconds, authenticating with username and password. Hint, just store the players username and password hash in a way the dataset fits into memory, so you avoid reading from disk and pick a programming language with good concurrency.

If you are taking hplus0603 advice and creates web api I do recommend this software (open source):

https://github.com/knutin/elli

I recently created a web api for authentication through my game servers myself, the code very short and hopefully scaleable. I have not yet tested the scability, but others probably have if you "google" it.

Memcached and Redis both can do hundreds of thousands of key-value reads per second from a large number of querying hosts.

If you do proper horizontal scalability of your web API servers, then the CPU efficiency of those machines don't *really* matter, as you can "just" add more of them, although having to add fewer is always better. Whether you write them in PHP, or Node, or Erlang, or C++, or J2EE, or Ruby, only adds a constant multiplier, assuming the use of the back-end systems is the same in each of the solutions.

Getting to 10,000 logins in a second is not that hard with a web-type API for batch requests, using this method. I just wish game simulation and other such real-time services followed the scalability rules of web APIs :-)
enum Bool { True, False, FileNotFound };

This topic is closed to new replies.

Advertisement