• Advertisement
Sign in to follow this  

Whats the best way to handle multiple servers?

This topic is 4794 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm curious about what is the accepted best way to handle multiple servers with a multi-player game? What I think of for this is all data from the clients is stored into one buffer then another process sends the data to the specific server that the data is meant for. The problem I have with this method is all communications are handled by a single process, possibly bogging down the system. Though this may not be a problem since redirecting the data packets to the correct server probably uses less processing than the work the server on the end has to do. Any thoughts?

Share this post


Link to post
Share on other sites
Advertisement
Up to a few thousand players, a gateway machine would probably be fine.

After that, you have to have the clients re-connect directly to the different servers, or to different gatekeepers for different servers. It should be noted that EverQuest did this already back in 2000.

One problem you'll have to solve is how to log in with username and password and authenticate only once, rather than having to re-authenticate with each server you connect to. I have a brief overview on my website that might help.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Latency comes in many separate stages, and a gateway will increas one of those stages. However, it's usually one of the smallest (or even the very smallest, if you're clever with your network design). Of course, several professional MMOG developers have been stupid and implemented naive gateways, which has helped earn them a bad name in some circles. Sigh.

An example latency path:

PC to cable modem: 0.5 ms
modem to ISP: 2 ms
ISP to MMOG (gateway) : 40ms
MMOG (gateway) to MMOG (other server) : 0.3 ms
MMOG (other server) parsing time: 0.1 ms
MMOG (other server) processing time: 4 ms
MMOG (other server) queue delay: 16 ms
MMOG (other server) compile + transmit response: 0.1 ms
return path all the way to PC: 42 ms

total = 105 ms
total added by gateway = 0.3 ms

Cluster-performance-design 101: never pre-empt the bottlenecks by saying "isn't X slow?". Always do the maths, and calculate the actual effect of X compared to the rest of the system :).

e.g. even if your gateway does it's own fat protocol processing, and takes a monumental 5 ms about it, the effect on the overall latency is indistinguishable.

PS: of course those figures are off the top of my head. Don't rely on them. They're approximately correct given all sorts of assumptions about the actual game design etc.

redmilamber

Share this post


Link to post
Share on other sites
You forgot one other big source of latency: The latency between the user pressing the key, and the program reading the key and queuing a network command, and the program actually coalescing network commands into packets. If your packet rate is 20 per second, that's 50 milliseconds max (25 average).

Btw: The latency between the modem and the actual Internet POP for your ISP is likely to be higher than 2 milliseconds.

Share this post


Link to post
Share on other sites
You can't go state-to-state in 0.3 milliseconds. The speed of light is slower than that!

If you have multiple server clusters in different geographic locations, and use gateways, I would recommend putting one gateway at each cluster, to avoid having to forward interactive data between geographically dispersed clusters.

Share this post


Link to post
Share on other sites
crap.. the speed of light is slower then that? so i guess we will never truly have a 0 ping system then, huh? how great it would be if we didnt have to worry about latency. throw in an infinite bandwith connection and remove the world of all cheaters and you have reached network programming heaven [grin].

actually, forget that. half the fun is dealing with this stuff [smile].

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by hplus0603
You forgot one other big source of latency: The latency between the user pressing the key, and the program reading the key and queuing a network command, and the program actually coalescing network commands into packets. If your packet rate is 20 per second, that's 50 milliseconds max (25 average).


:) very good point. I'm too accustomed to doing FPS designs, where the rule of thumbs are all measured in RTT from "message starts to be transmitted from client's NIC" until "message received at client NIC".

Quote:

Btw: The latency between the modem and the actual Internet POP for your ISP is likely to be higher than 2 milliseconds.


I was going for a lower bound there, to minimize the total latency calc, since it seems to depend a great deal on geographic location. Here in the UK a lot of broadband is at that level. I've had two providers in a row who have consistently been sub-5-milliseconds. Where I am today (out in the middle of rural england, no more than 5 houses within 5 miles of me) I'm getting 15 ms.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by hplus0603
You can't go state-to-state in 0.3 milliseconds. The speed of light is slower than that!


Exactly...I thought it would be obvious that a sub-1-ms time was going to have to be on a local LAN :). So, as hplus suggests, I was assuming that you had a gateway per geoggraphically located cluster (IMHO anything else is insane, in this context. You can have a front-end that lets the user select a gateway, which is itself a sort-of-pseudo-gateway (it might automatically redirect to a gateway based on netblock of client, for instnace), but as far as these low-level gatewaus go they HAVE to sit in their cluster).

@ graveyard:

ping time for light to cross atlantic ocean == 40 ms (20 + 20).

However...there are ways of going much faster than light. There is a quantum system that has been demonstrated to transfer information much faster (reasonably well accepted) and another that seems to suggest actual matter travelled faster than light (not so well accepted).

It is not unreasonable to expect that a quantum connection will eventually be setup across each inter-continent link. "Eventually" could well be 50 years away.

There are 3 invariants in the world of computers:
- latency never decreases (compared to bandwidth and processing power)
- batteries never get smaller (compared to microchips)
- power cannot be supplied wirelessly (without frying every poor soul who walks through the space the power is moving through)

All could be beaten some day, but have held true for almost as long as Moore's law. Look at hard-disk latencies today and compare them to 10 years ago. Do the same for transfer rates :). Ditto with networking. Ditto with system bus etc.

redmilamber

Share this post


Link to post
Share on other sites
Btw: that 40 ms ping time is the best possible case, where someone in New England is pinging someone in Old England. In reality, it'll typically be worse:


traceroute to nada.kth.se (130.237.222.80), 30 hops max, 38 byte packets
1 er1.sfo1.speakeasy.net (69.17.45.1) 17.556 ms 24.354 ms 17.552 ms
2 220.ge-0-1-0.cr2.sfo1.speakeasy.net (69.17.83.177) 18.195 ms 18.067 ms 19.432 ms
3 ge-4-0-440.ipcolo1.SanJose1.Level3.net (209.247.156.221) 18.497 ms 14.225 ms 16.655 ms
4 ae-1-55.bbr1.SanJose1.Level3.net (4.68.123.129) 16.409 ms 15.193 ms 16.904 ms
5 so-0-1-0.bbr1.NewYork1.Level3.net (64.159.1.41) 99.396 ms 91.502 ms 88.806 ms
6 4.68.128.105 (4.68.128.105) 155.533 ms 153.322 ms 154.812 ms
7 so-0-1-0.mp2.Stockholm1.Level3.net (4.68.128.70) 189.250 ms 188.385 ms 188.182 ms
8 so-11-0.hsa2.Stockholm1.Level3.net (213.242.68.206) 187.163 ms 188.593 ms 187.281 ms
9 213.242.69.22 (213.242.69.22) 187.008 ms 187.002 ms 186.154 ms
10 stockholm2-POS6-0.sunet.se (193.10.252.178) 187.628 ms 187.369 ms 258.679 ms
11 stockholm4-POS0.sunet.se (130.242.82.50) 186.858 ms 186.774 ms 186.722 ms
12 kth2-srp1.sunet.se (130.242.85.68) 187.420 ms * 186.383 ms
13 ea4-kth2-b.gw.kth.se (130.237.211.242) 187.131 ms 193.494 ms 193.076 ms
14 nada-ea4-p2p.gw.kth.se (130.237.211.210) 190.505 ms 188.472 ms 188.914 ms
15 faun.nada.kth.se (130.237.222.80) 187.850 ms 188.345 ms 187.890 ms


San Jose, CA to New York, NY is about 75 milliseconds; New York to Europe is an additional 65 or so. No doubt a bit of that is transmission framing and queuing delays, plus the fact that cables don't run straight. You'll also note that my DSL circuit adds 17 ms -- and I live in the middle of Silicon Valley! We always get early technology, and thus we have the worst actual implementations. Just like NTSC TV ;-)

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement