Client-Server (Centralized) Topology for an MMO
this is kind of off topic, but i didn't want to start a whole new thread for it.... is it possible to set up a world that could scale so that two players standing next to each other could actually be connected to different machines? if this were possible, it would allow to do things like have massive battles in MMORPG's. my guess is it's probably possible but just wouldn't be worth the expenses to get it to work. if anyone has ever played Shadowbane, this game suffered from not being able to scale like that. i played in the begining though, so it might have changed by now, but the jist of the main point of the game was to be in a guild and go to war and destroy other peoples cities... the only problem though was when lots of people all crowded around in one spot, it was in _super_ slow motion.
Author
Good to see this thread got some more responses. Thanks guys.
I actually own Programming Gems 1-4, and tons of other books, as you can imagine I can't get through them all, though someday I plan to ;). I looked through the first one, didn't see anything about networking, and assumed the rest would lack it as well. Thanks for the tip!
If I can manage 2000 players on a single server, without having to have seperate servers for chat, login, etc. then I'd rather do without it (KISS - why add complexity when it's not needed). Our project is very ambitious, but we don't want more then 1000 players in a single gameworld. Even if we do acheive that kind of success (over the server limit), opening up another server (seperate gameworld) would be easy enough.
Although with a chat server and login server this is probably true, I tend to think that a seperate server to run AI routines would have a good reason. Being able to run complex AI routines (optimal path-planning, complex behavior with logic systems, etc.) might be able to make good use of the extra hardware. But then again this could be, like I mentioned earlier, adding complexity where it's not needed. You can get away with "good enough" AI in most cases, and it need not be optimal.
Just curious, why would guild chat be an n-squared communications bottleneck? It's very late, so it might be obvious, but still I don't understand it :).
I think the problem is this:
If you are standing right next to someone, you need to be sent all the information about them (location, what they look like, what actions they are performing, etc). Now the only way for you to recieve this information is to get it from the server you are connected to. Thus, that person needs to be connected to the same server.
Either that or the server you are talking to has to request the server they are talking to send you that information (how does it know this? every server will have to know who is connected to each other server). Seems to get complex quickly.
Wow I'm tired.
Night folks.
Quote:Original post by Anonymous Poster
Look in Game Programming Gems 4 (thousands of clients per server) if you can find a copy of the book.
I actually own Programming Gems 1-4, and tons of other books, as you can imagine I can't get through them all, though someday I plan to ;). I looked through the first one, didn't see anything about networking, and assumed the rest would lack it as well. Thanks for the tip!
Quote:Original post by Anonymous Poster
Easy to develop, crap performance. Don't go there if you want thousands of players. Fine if you only have a small game (<3,000 players)
If I can manage 2000 players on a single server, without having to have seperate servers for chat, login, etc. then I'd rather do without it (KISS - why add complexity when it's not needed). Our project is very ambitious, but we don't want more then 1000 players in a single gameworld. Even if we do acheive that kind of success (over the server limit), opening up another server (seperate gameworld) would be easy enough.
Quote:Because if you split up the duties to a bunch of individual servers you introduce mass latency issues and you are killing your performance for basically no reason.
Although with a chat server and login server this is probably true, I tend to think that a seperate server to run AI routines would have a good reason. Being able to run complex AI routines (optimal path-planning, complex behavior with logic systems, etc.) might be able to make good use of the extra hardware. But then again this could be, like I mentioned earlier, adding complexity where it's not needed. You can get away with "good enough" AI in most cases, and it need not be optimal.
Quote:If you carefully design and implement a split-server system, you can build a system that scales linearly by just adding hardware, basically until you hit some unforeseen n-squared communications bottleneck (guild chat, anyone?)
Just curious, why would guild chat be an n-squared communications bottleneck? It's very late, so it might be obvious, but still I don't understand it :).
Quote:is it possible to set up a world that could scale so that two players standing next to each other could actually be connected to different machines? if this were possible, it would allow to do things like have massive battles in MMORPG's. my guess is it's probably possible but just wouldn't be worth the expenses to get it to work. if anyone has ever played Shadowbane, this game suffered from not being able to scale like that.
I think the problem is this:
If you are standing right next to someone, you need to be sent all the information about them (location, what they look like, what actions they are performing, etc). Now the only way for you to recieve this information is to get it from the server you are connected to. Thus, that person needs to be connected to the same server.
Either that or the server you are talking to has to request the server they are talking to send you that information (how does it know this? every server will have to know who is connected to each other server). Seems to get complex quickly.
Wow I'm tired.
Night folks.
well, i guess i was thinking somehthing like this... you have a "master server" whos only job is to act as a router for communicating between the client and the server he is connected to. this router chose which players were assigned to what machine. now, the game would still be partitioned based on zones. when the player logs on, the router server assignes them to a machine, and then tells every machine who is currently controlling a client in that zone about this event, and the data with it. this is how everything would work. client asks to move, server receives it, tells the router guy who tells the other servers who tells their clients. is the latency of the communication from one server to the other so bad even if they are all right next to each other on the same network? with this scheme, wouldnt you be able to scale it so you could have hundreds of people packed into a tight spot and it would be just as if they were spread apart? my guess is currently this is unrealistic since it would be a waste of bandwith, but maybe one day when bandwith / hardware is cheaper, we could have this scheme to allow not only massive battles in MMORPG's but also realistic physics as well. the only problem is that master server would be under load, so we would have to distribute that work somehow as well. hplus, i think we have talked about this before but i can't remember the outcome of our convo [grin].
In our system, you will run into the case that a guy standing next to you is "homed" on another simulation server than you, and the system keeps working. One of the many hard problems we spent five years to solve :-) We scale linearly by plugging in more hardware, for many scalability scenarios.
Why did I suggest guild chat might be O(N^2)? Well, here's a possible construction. I'm using the name "server" to mean physical machin within a "cluster" of machines, which work together to run a "shard" or instance of the world (in unified-world systems, there's only one shard):
- To linearly scale with volume, you need to assign each player some server to send chat to.
- This server needs to forward the chat to the appropriate channels, which may be a /tell (any other chat server could be the recipient), a /local or /shout (any player in the same general area as the player could be the recipient), or something like /guild (any member of the guild, on any current chat server, could be the recipient).
- From the player's chat recipient server's point of view, you can either establish connections to destination for each message you send, and then tear them down when the message is sent, or you can pre-establish connections up front.
- If you keep establishing and tearing down connections (even with LRU replacement), you'll thrash your networking stack pretty heavily, and have high latencies on chat propagation.
- If you pre-establish communications, then each chat server has to talk to each other chat server. This is an N^2 factor in your scalability equation.
- There's also the distributed database question of how to find a specific player that's the target of a /tell command, or specific players that are close enough for a /local (nearby) send.
Depending on how many players you can put on the chat server, how you decide to allocate players over chat servers, whether you allocate special channels per guild/zone or not, you may be able to push off the N^2 such that, at 100,000 players, N is something like 10, and it just isn't much of a problem. This would be the case if each chat server did only chat. In another extreme, each server might be responsible for everything that happens to players connected to that server (chat, persistency, simulation, trade, ...); then the N in the case of 100,000 players might be closer to 1000, and the N^2 portion of the equation will start looking really bad.
If you have distinct zones like EverQuest, you can route local/zone chat through there. If you limit the number of guilds a player can be in to a small number, you can allocate guilds to servers and create a limited number of connections for this. If you can accept a high latency for /tells, you can create connectivity on demand, or use a central tell distribution server that's start-connected to other chat recipient servers. Meanwhile, if you want low latency, seamless world, and no limit on number of guilds per member, you need a very different design.
Meanwhile, if you want < 2,000 players, you can just stick 'em all on a single server and call it a day ;-)
Why did I suggest guild chat might be O(N^2)? Well, here's a possible construction. I'm using the name "server" to mean physical machin within a "cluster" of machines, which work together to run a "shard" or instance of the world (in unified-world systems, there's only one shard):
- To linearly scale with volume, you need to assign each player some server to send chat to.
- This server needs to forward the chat to the appropriate channels, which may be a /tell (any other chat server could be the recipient), a /local or /shout (any player in the same general area as the player could be the recipient), or something like /guild (any member of the guild, on any current chat server, could be the recipient).
- From the player's chat recipient server's point of view, you can either establish connections to destination for each message you send, and then tear them down when the message is sent, or you can pre-establish connections up front.
- If you keep establishing and tearing down connections (even with LRU replacement), you'll thrash your networking stack pretty heavily, and have high latencies on chat propagation.
- If you pre-establish communications, then each chat server has to talk to each other chat server. This is an N^2 factor in your scalability equation.
- There's also the distributed database question of how to find a specific player that's the target of a /tell command, or specific players that are close enough for a /local (nearby) send.
Depending on how many players you can put on the chat server, how you decide to allocate players over chat servers, whether you allocate special channels per guild/zone or not, you may be able to push off the N^2 such that, at 100,000 players, N is something like 10, and it just isn't much of a problem. This would be the case if each chat server did only chat. In another extreme, each server might be responsible for everything that happens to players connected to that server (chat, persistency, simulation, trade, ...); then the N in the case of 100,000 players might be closer to 1000, and the N^2 portion of the equation will start looking really bad.
If you have distinct zones like EverQuest, you can route local/zone chat through there. If you limit the number of guilds a player can be in to a small number, you can allocate guilds to servers and create a limited number of connections for this. If you can accept a high latency for /tells, you can create connectivity on demand, or use a central tell distribution server that's start-connected to other chat recipient servers. Meanwhile, if you want low latency, seamless world, and no limit on number of guilds per member, you need a very different design.
Meanwhile, if you want < 2,000 players, you can just stick 'em all on a single server and call it a day ;-)
My approach:
One login server, players connect to this, and this connects to 'zone servers', thereby essentially just being a proxy server.
Each zone server handles updates for that particular area, whether it be an instanced version of a dungeon, or just a section of the game world.
When a person leaves one zone (or is about to) and enters another, the connection is passed from one zone server to another. In the case of zone loading (EQ), this is fairly straightforward. In the case of 'hidden' zone loading (Camelot), this requires some duplicate information (or, the login server is connected to multiple zone servers so that people in range in both zones see and are seen by the zoning character).
The aspect about a huge number of people in one area isn't really an issue of server hardware keeping up with things. Sure, it can be a bottleneck, but getting update information for 500 people constantly over a 1 hour battle makes up for a lot of packets. Historically, games haven't handled this because of graphics (can't render that much stuff on screen), and bandwidth (can't receive that many updates that fast). Historically, solutions have been toned down graphics and network usage (as well as prioritized packets - Send positions, then effects, then animations, then clothing, etc).
Now on the server side, having your server farm with multiple servers handling one zone, while doable, adds levels of complexity that just make life more difficult. But given the above approach, you could also use two servers for one zone just by load balancing connections (Server 1 processes 250 player updates, Server 2 the other 250), but you still end up with having to send updates to the same 500 people.
-Alamar
One login server, players connect to this, and this connects to 'zone servers', thereby essentially just being a proxy server.
Each zone server handles updates for that particular area, whether it be an instanced version of a dungeon, or just a section of the game world.
When a person leaves one zone (or is about to) and enters another, the connection is passed from one zone server to another. In the case of zone loading (EQ), this is fairly straightforward. In the case of 'hidden' zone loading (Camelot), this requires some duplicate information (or, the login server is connected to multiple zone servers so that people in range in both zones see and are seen by the zoning character).
The aspect about a huge number of people in one area isn't really an issue of server hardware keeping up with things. Sure, it can be a bottleneck, but getting update information for 500 people constantly over a 1 hour battle makes up for a lot of packets. Historically, games haven't handled this because of graphics (can't render that much stuff on screen), and bandwidth (can't receive that many updates that fast). Historically, solutions have been toned down graphics and network usage (as well as prioritized packets - Send positions, then effects, then animations, then clothing, etc).
Now on the server side, having your server farm with multiple servers handling one zone, while doable, adds levels of complexity that just make life more difficult. But given the above approach, you could also use two servers for one zone just by load balancing connections (Server 1 processes 250 player updates, Server 2 the other 250), but you still end up with having to send updates to the same 500 people.
-Alamar
I think most major MMO have several "servers" for different functions; a login server, NPC server, player server, and a download server. I know that Lineage 2 (which I am currently playing) has at least some form of an NPC server that handles path finding, shops, combat, drops, etc. I'm not sure if its simply threaded, or if its actually another server code. I'd probably take a stab at it and say that its just a different thread, not a different process.
I've always been interested in networking, so in my many brianstorms I always seem to write down network ideas and solutions to things I may never get around to programming (or working on a project). Its just how I teach myself, may be a bad method, but thats how I do it. The design that I had going was three seperate servers; account login server, game logic server, and bittorrent file/tracker.
City of Heroes has an instance server which handles the small little "battle" scenes for quests that groups go into. They need this because there are several groups at the same time sometimes doing the same quest, in the same zone, in the same area. They have instances in order to allow people to do the quests at the same time. I think World of Warcraft has something similar (I've been told) but I've never played it.
I've always been interested in networking, so in my many brianstorms I always seem to write down network ideas and solutions to things I may never get around to programming (or working on a project). Its just how I teach myself, may be a bad method, but thats how I do it. The design that I had going was three seperate servers; account login server, game logic server, and bittorrent file/tracker.
City of Heroes has an instance server which handles the small little "battle" scenes for quests that groups go into. They need this because there are several groups at the same time sometimes doing the same quest, in the same zone, in the same area. They have instances in order to allow people to do the quests at the same time. I think World of Warcraft has something similar (I've been told) but I've never played it.
Quote:Original post by hplus0603
Meanwhile, if you want < 2,000 players, you can just stick 'em all on a single server and call it a day ;-)
I agree.. although I would say depending on the problem space the numbers can be somewhat higher. With RunUO the C# Ultima Online emulator it has been tested to have 3200 concurrent TCP connections playing.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
