# Whats the best way to handle multiple servers?

This topic is 4794 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I'm curious about what is the accepted best way to handle multiple servers with a multi-player game? What I think of for this is all data from the clients is stored into one buffer then another process sends the data to the specific server that the data is meant for. The problem I have with this method is all communications are handled by a single process, possibly bogging down the system. Though this may not be a problem since redirecting the data packets to the correct server probably uses less processing than the work the server on the end has to do. Any thoughts?

##### Share on other sites
Up to a few thousand players, a gateway machine would probably be fine.

After that, you have to have the clients re-connect directly to the different servers, or to different gatekeepers for different servers. It should be noted that EverQuest did this already back in 2000.

One problem you'll have to solve is how to log in with username and password and authenticate only once, rather than having to re-authenticate with each server you connect to. I have a brief overview on my website that might help.

##### Share on other sites
Latency comes in many separate stages, and a gateway will increas one of those stages. However, it's usually one of the smallest (or even the very smallest, if you're clever with your network design). Of course, several professional MMOG developers have been stupid and implemented naive gateways, which has helped earn them a bad name in some circles. Sigh.

An example latency path:

PC to cable modem: 0.5 ms
modem to ISP: 2 ms
ISP to MMOG (gateway) : 40ms
MMOG (gateway) to MMOG (other server) : 0.3 ms
MMOG (other server) parsing time: 0.1 ms
MMOG (other server) processing time: 4 ms
MMOG (other server) queue delay: 16 ms
MMOG (other server) compile + transmit response: 0.1 ms
return path all the way to PC: 42 ms

total = 105 ms
total added by gateway = 0.3 ms

Cluster-performance-design 101: never pre-empt the bottlenecks by saying "isn't X slow?". Always do the maths, and calculate the actual effect of X compared to the rest of the system :).

e.g. even if your gateway does it's own fat protocol processing, and takes a monumental 5 ms about it, the effect on the overall latency is indistinguishable.

PS: of course those figures are off the top of my head. Don't rely on them. They're approximately correct given all sorts of assumptions about the actual game design etc.

redmilamber

##### Share on other sites
You forgot one other big source of latency: The latency between the user pressing the key, and the program reading the key and queuing a network command, and the program actually coalescing network commands into packets. If your packet rate is 20 per second, that's 50 milliseconds max (25 average).

Btw: The latency between the modem and the actual Internet POP for your ISP is likely to be higher than 2 milliseconds.

##### Share on other sites
are these numbers only accurate if the 2 servers are sitting next to each other? or will these still hold up if they are not in the same building, or state even?

##### Share on other sites
You can't go state-to-state in 0.3 milliseconds. The speed of light is slower than that!

If you have multiple server clusters in different geographic locations, and use gateways, I would recommend putting one gateway at each cluster, to avoid having to forward interactive data between geographically dispersed clusters.

##### Share on other sites
crap.. the speed of light is slower then that? so i guess we will never truly have a 0 ping system then, huh? how great it would be if we didnt have to worry about latency. throw in an infinite bandwith connection and remove the world of all cheaters and you have reached network programming heaven [grin].

actually, forget that. half the fun is dealing with this stuff [smile].

##### Share on other sites
Quote:
 Original post by hplus0603You forgot one other big source of latency: The latency between the user pressing the key, and the program reading the key and queuing a network command, and the program actually coalescing network commands into packets. If your packet rate is 20 per second, that's 50 milliseconds max (25 average).

:) very good point. I'm too accustomed to doing FPS designs, where the rule of thumbs are all measured in RTT from "message starts to be transmitted from client's NIC" until "message received at client NIC".

Quote:
 Btw: The latency between the modem and the actual Internet POP for your ISP is likely to be higher than 2 milliseconds.

I was going for a lower bound there, to minimize the total latency calc, since it seems to depend a great deal on geographic location. Here in the UK a lot of broadband is at that level. I've had two providers in a row who have consistently been sub-5-milliseconds. Where I am today (out in the middle of rural england, no more than 5 houses within 5 miles of me) I'm getting 15 ms.

##### Share on other sites
Quote:
 Original post by hplus0603You can't go state-to-state in 0.3 milliseconds. The speed of light is slower than that!

Exactly...I thought it would be obvious that a sub-1-ms time was going to have to be on a local LAN :). So, as hplus suggests, I was assuming that you had a gateway per geoggraphically located cluster (IMHO anything else is insane, in this context. You can have a front-end that lets the user select a gateway, which is itself a sort-of-pseudo-gateway (it might automatically redirect to a gateway based on netblock of client, for instnace), but as far as these low-level gatewaus go they HAVE to sit in their cluster).

@ graveyard:

ping time for light to cross atlantic ocean == 40 ms (20 + 20).

However...there are ways of going much faster than light. There is a quantum system that has been demonstrated to transfer information much faster (reasonably well accepted) and another that seems to suggest actual matter travelled faster than light (not so well accepted).

It is not unreasonable to expect that a quantum connection will eventually be setup across each inter-continent link. "Eventually" could well be 50 years away.

There are 3 invariants in the world of computers:
- latency never decreases (compared to bandwidth and processing power)
- batteries never get smaller (compared to microchips)
- power cannot be supplied wirelessly (without frying every poor soul who walks through the space the power is moving through)

All could be beaten some day, but have held true for almost as long as Moore's law. Look at hard-disk latencies today and compare them to 10 years ago. Do the same for transfer rates :). Ditto with networking. Ditto with system bus etc.

redmilamber

##### Share on other sites
Btw: that 40 ms ping time is the best possible case, where someone in New England is pinging someone in Old England. In reality, it'll typically be worse:

traceroute to nada.kth.se (130.237.222.80), 30 hops max, 38 byte packets 1  er1.sfo1.speakeasy.net (69.17.45.1)  17.556 ms  24.354 ms  17.552 ms 2  220.ge-0-1-0.cr2.sfo1.speakeasy.net (69.17.83.177)  18.195 ms  18.067 ms  19.432 ms 3  ge-4-0-440.ipcolo1.SanJose1.Level3.net (209.247.156.221)  18.497 ms  14.225 ms  16.655 ms 4  ae-1-55.bbr1.SanJose1.Level3.net (4.68.123.129)  16.409 ms  15.193 ms  16.904 ms 5  so-0-1-0.bbr1.NewYork1.Level3.net (64.159.1.41)  99.396 ms  91.502 ms  88.806 ms 6  4.68.128.105 (4.68.128.105)  155.533 ms  153.322 ms  154.812 ms 7  so-0-1-0.mp2.Stockholm1.Level3.net (4.68.128.70)  189.250 ms  188.385 ms  188.182 ms 8  so-11-0.hsa2.Stockholm1.Level3.net (213.242.68.206)  187.163 ms  188.593 ms  187.281 ms 9  213.242.69.22 (213.242.69.22)  187.008 ms  187.002 ms  186.154 ms10  stockholm2-POS6-0.sunet.se (193.10.252.178)  187.628 ms  187.369 ms  258.679 ms11  stockholm4-POS0.sunet.se (130.242.82.50)  186.858 ms  186.774 ms  186.722 ms12  kth2-srp1.sunet.se (130.242.85.68)  187.420 ms *  186.383 ms13  ea4-kth2-b.gw.kth.se (130.237.211.242)  187.131 ms  193.494 ms  193.076 ms14  nada-ea4-p2p.gw.kth.se (130.237.211.210)  190.505 ms  188.472 ms  188.914 ms15  faun.nada.kth.se (130.237.222.80)  187.850 ms  188.345 ms  187.890 ms

San Jose, CA to New York, NY is about 75 milliseconds; New York to Europe is an additional 65 or so. No doubt a bit of that is transmission framing and queuing delays, plus the fact that cables don't run straight. You'll also note that my DSL circuit adds 17 ms -- and I live in the middle of Silicon Valley! We always get early technology, and thus we have the worst actual implementations. Just like NTSC TV ;-)