Sign in to follow this  

Client-Server (Centralized) Topology for an MMO

This topic is 4663 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm currently researching the viability of a centralized server networking model for an MMO, and looking into possible alternatives. I've found one article on GDNet: Distributed Gaming I would be curious if anyone has any information on the subject (both centralized and alternative topologies) they would be willing to share. I'm also looking for information on how it is possible to have multiple servers handle different elements of the game (IE having a seperate login server, chat server, etc.), though I suppose that wouldn't be too complicated to develop by simply sending specific messages to different servers who all have access to the same database. Anyway, if you've got any links or info you think would be helpful to my research, I'd greatly apreciate it. Thanks.

Share this post


Link to post
Share on other sites
In our system we use something called a NPC client. This is a seperate application from the main server that logs in and is assigned a group of NPC's to control. This can be things like path wandering, and combat behaviours.

This way we can scale up the number of NPCs and their complexity by simply adding more npcclients. For example, a big mob may have one client by itself while 200 rats in a sewer can be all be handled by another one. The server and npcclient communicate over the network and the npcclient is notified of information it needs like who is attacking it, where items are, etc.

Share this post


Link to post
Share on other sites
Interesting acraig, I hadn't thought of that.

Is your system being used in practice?
I'm curious on the ratio of NPCClients to NPCs you have. I would think a physical server given decent specs could handle a good amount of AI entity behaviors, unless very complex.

Also, how does this NPCClient interact with the main server?
I'm guessing it grinds out all the AI details and sends the results to the main server (ie given a current and target location, the main server recieves the "updated position" of the npc after the pathfinding routine is run).

Are there any major pros/cons to this system you've found?

Thanks for the reply.

Share this post


Link to post
Share on other sites
Quote:
Original post by x_gamer_x
Interesting acraig, I hadn't thought of that.

Is your system being used in practice?
I'm curious on the ratio of NPCClients to NPCs you have. I would think a physical server given decent specs could handle a good amount of AI entity behaviors, unless very complex.


Yes, is being used in our PlaneShift project to control the NPCs. Right now we are only using the one npcclient to control all the npcs. There is around 150-200 NPCs ( this includes creatures like rats and the like ). It handles character wandering around and their combat 'brains'. At the moment CPU usage is very low.

Quote:

Also, how does this NPCClient interact with the main server?
I'm guessing it grinds out all the AI details and sends the results to the main server (ie given a current and target location, the main server recieves the "updated position" of the npc after the pathfinding routine is run).

Pretty much yes. The behaviours are scripted XML files that the npc client loads ( things like Wanderer script etc ). The NPCclients then figure out which of it's behaviours have priority for which mob and does the calculations and sends the results back to the server. The server makes sure the npcclient has updated data by sending it 'perceptions'. For example if a player attacks a rat the server will send an 'attack' perception to the npcclient responsible for that NPC. The npcclient will then deal with that perception, most likely will push the attack behaviour to the top of it's behaviour queue.

Quote:

Are there any major pros/cons to this system you've found?


The major pro is that we can offload some of the required CPU power off of the main server. Since we use donated hardware it makes it easier to make use of them. The con is that it will take up some of the bandwidth, so many npcclients make steal a lot of the network.

You can see our npcclient source code here

Share this post


Link to post
Share on other sites
Thanks again for the quick response, and wow, I wasn't expecting to see source code :). Rating++.

I don't have time to go over it right now, but I definetly plan to in the future.

If anyone else has information on the suject, I'm still looking ;).

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by x_gamer_x
I would be curious if anyone has any information on the subject


Look in Game Programming Gems 4 (thousands of clients per server) if you can find a copy of the book.

Quote:

(both centralized and alternative topologies) they would be willing to share. I'm also looking for information on how it is possible to have multiple servers handle different elements of the game (IE having a seperate login server, chat server, etc.), though I suppose that wouldn't be too complicated to develop


Easy to develop, crap performance. Don't go there if you want thousands of players. Fine if you only have a small game (<3,000 players)

Share this post


Link to post
Share on other sites
just curious, but why is it crap performance to split up duties into individual servers? i had always assumed this is how commercial games did it.. if not, how can it scale to handle many players?

Share this post


Link to post
Share on other sites
Quote:
Original post by graveyard filla
just curious, but why is it crap performance to split up duties into individual servers?

Because if you split up the duties to a bunch of individual servers you introduce mass latency issues and you are killing your performance for basically no reason.

Share this post


Link to post
Share on other sites
so, how does it work then? some sort of system where all machines share the same set of jobs? then scaling is just a matter of adding more machines? even then, the commucation between the servers is nessasary i would think...

Share this post


Link to post
Share on other sites
Quote:
if you split up the duties to a bunch of individual servers you introduce mass latency issues and you are killing your performance for basically no reason


If you do a poor job of factoring your servers, that's certainly true.

If you carefully design and implement a split-server system, you can build a system that scales linearly by just adding hardware, basically until you hit some unforeseen n-squared communications bottleneck (guild chat, anyone?) or run out of network backplane bandwidth (unlikely with current big iron hardware available).

Share this post


Link to post
Share on other sites
this is kind of off topic, but i didn't want to start a whole new thread for it.... is it possible to set up a world that could scale so that two players standing next to each other could actually be connected to different machines? if this were possible, it would allow to do things like have massive battles in MMORPG's. my guess is it's probably possible but just wouldn't be worth the expenses to get it to work. if anyone has ever played Shadowbane, this game suffered from not being able to scale like that. i played in the begining though, so it might have changed by now, but the jist of the main point of the game was to be in a guild and go to war and destroy other peoples cities... the only problem though was when lots of people all crowded around in one spot, it was in _super_ slow motion.

Share this post


Link to post
Share on other sites
Good to see this thread got some more responses. Thanks guys.

Quote:
Original post by Anonymous Poster

Look in Game Programming Gems 4 (thousands of clients per server) if you can find a copy of the book.


I actually own Programming Gems 1-4, and tons of other books, as you can imagine I can't get through them all, though someday I plan to ;). I looked through the first one, didn't see anything about networking, and assumed the rest would lack it as well. Thanks for the tip!

Quote:
Original post by Anonymous Poster
Easy to develop, crap performance. Don't go there if you want thousands of players. Fine if you only have a small game (<3,000 players)


If I can manage 2000 players on a single server, without having to have seperate servers for chat, login, etc. then I'd rather do without it (KISS - why add complexity when it's not needed). Our project is very ambitious, but we don't want more then 1000 players in a single gameworld. Even if we do acheive that kind of success (over the server limit), opening up another server (seperate gameworld) would be easy enough.

Quote:
Because if you split up the duties to a bunch of individual servers you introduce mass latency issues and you are killing your performance for basically no reason.


Although with a chat server and login server this is probably true, I tend to think that a seperate server to run AI routines would have a good reason. Being able to run complex AI routines (optimal path-planning, complex behavior with logic systems, etc.) might be able to make good use of the extra hardware. But then again this could be, like I mentioned earlier, adding complexity where it's not needed. You can get away with "good enough" AI in most cases, and it need not be optimal.

Quote:
If you carefully design and implement a split-server system, you can build a system that scales linearly by just adding hardware, basically until you hit some unforeseen n-squared communications bottleneck (guild chat, anyone?)


Just curious, why would guild chat be an n-squared communications bottleneck? It's very late, so it might be obvious, but still I don't understand it :).

Quote:
is it possible to set up a world that could scale so that two players standing next to each other could actually be connected to different machines? if this were possible, it would allow to do things like have massive battles in MMORPG's. my guess is it's probably possible but just wouldn't be worth the expenses to get it to work. if anyone has ever played Shadowbane, this game suffered from not being able to scale like that.

I think the problem is this:
If you are standing right next to someone, you need to be sent all the information about them (location, what they look like, what actions they are performing, etc). Now the only way for you to recieve this information is to get it from the server you are connected to. Thus, that person needs to be connected to the same server.

Either that or the server you are talking to has to request the server they are talking to send you that information (how does it know this? every server will have to know who is connected to each other server). Seems to get complex quickly.

Wow I'm tired.

Night folks.

Share this post


Link to post
Share on other sites
well, i guess i was thinking somehthing like this... you have a "master server" whos only job is to act as a router for communicating between the client and the server he is connected to. this router chose which players were assigned to what machine. now, the game would still be partitioned based on zones. when the player logs on, the router server assignes them to a machine, and then tells every machine who is currently controlling a client in that zone about this event, and the data with it. this is how everything would work. client asks to move, server receives it, tells the router guy who tells the other servers who tells their clients. is the latency of the communication from one server to the other so bad even if they are all right next to each other on the same network? with this scheme, wouldnt you be able to scale it so you could have hundreds of people packed into a tight spot and it would be just as if they were spread apart? my guess is currently this is unrealistic since it would be a waste of bandwith, but maybe one day when bandwith / hardware is cheaper, we could have this scheme to allow not only massive battles in MMORPG's but also realistic physics as well. the only problem is that master server would be under load, so we would have to distribute that work somehow as well. hplus, i think we have talked about this before but i can't remember the outcome of our convo [grin].

Share this post


Link to post
Share on other sites
In our system, you will run into the case that a guy standing next to you is "homed" on another simulation server than you, and the system keeps working. One of the many hard problems we spent five years to solve :-) We scale linearly by plugging in more hardware, for many scalability scenarios.

Why did I suggest guild chat might be O(N^2)? Well, here's a possible construction. I'm using the name "server" to mean physical machin within a "cluster" of machines, which work together to run a "shard" or instance of the world (in unified-world systems, there's only one shard):

- To linearly scale with volume, you need to assign each player some server to send chat to.
- This server needs to forward the chat to the appropriate channels, which may be a /tell (any other chat server could be the recipient), a /local or /shout (any player in the same general area as the player could be the recipient), or something like /guild (any member of the guild, on any current chat server, could be the recipient).
- From the player's chat recipient server's point of view, you can either establish connections to destination for each message you send, and then tear them down when the message is sent, or you can pre-establish connections up front.
- If you keep establishing and tearing down connections (even with LRU replacement), you'll thrash your networking stack pretty heavily, and have high latencies on chat propagation.
- If you pre-establish communications, then each chat server has to talk to each other chat server. This is an N^2 factor in your scalability equation.
- There's also the distributed database question of how to find a specific player that's the target of a /tell command, or specific players that are close enough for a /local (nearby) send.

Depending on how many players you can put on the chat server, how you decide to allocate players over chat servers, whether you allocate special channels per guild/zone or not, you may be able to push off the N^2 such that, at 100,000 players, N is something like 10, and it just isn't much of a problem. This would be the case if each chat server did only chat. In another extreme, each server might be responsible for everything that happens to players connected to that server (chat, persistency, simulation, trade, ...); then the N in the case of 100,000 players might be closer to 1000, and the N^2 portion of the equation will start looking really bad.

If you have distinct zones like EverQuest, you can route local/zone chat through there. If you limit the number of guilds a player can be in to a small number, you can allocate guilds to servers and create a limited number of connections for this. If you can accept a high latency for /tells, you can create connectivity on demand, or use a central tell distribution server that's start-connected to other chat recipient servers. Meanwhile, if you want low latency, seamless world, and no limit on number of guilds per member, you need a very different design.

Meanwhile, if you want < 2,000 players, you can just stick 'em all on a single server and call it a day ;-)

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
My approach:

One login server, players connect to this, and this connects to 'zone servers', thereby essentially just being a proxy server.

Each zone server handles updates for that particular area, whether it be an instanced version of a dungeon, or just a section of the game world.

When a person leaves one zone (or is about to) and enters another, the connection is passed from one zone server to another. In the case of zone loading (EQ), this is fairly straightforward. In the case of 'hidden' zone loading (Camelot), this requires some duplicate information (or, the login server is connected to multiple zone servers so that people in range in both zones see and are seen by the zoning character).

The aspect about a huge number of people in one area isn't really an issue of server hardware keeping up with things. Sure, it can be a bottleneck, but getting update information for 500 people constantly over a 1 hour battle makes up for a lot of packets. Historically, games haven't handled this because of graphics (can't render that much stuff on screen), and bandwidth (can't receive that many updates that fast). Historically, solutions have been toned down graphics and network usage (as well as prioritized packets - Send positions, then effects, then animations, then clothing, etc).

Now on the server side, having your server farm with multiple servers handling one zone, while doable, adds levels of complexity that just make life more difficult. But given the above approach, you could also use two servers for one zone just by load balancing connections (Server 1 processes 250 player updates, Server 2 the other 250), but you still end up with having to send updates to the same 500 people.

-Alamar

Share this post


Link to post
Share on other sites
I think most major MMO have several "servers" for different functions; a login server, NPC server, player server, and a download server. I know that Lineage 2 (which I am currently playing) has at least some form of an NPC server that handles path finding, shops, combat, drops, etc. I'm not sure if its simply threaded, or if its actually another server code. I'd probably take a stab at it and say that its just a different thread, not a different process.

I've always been interested in networking, so in my many brianstorms I always seem to write down network ideas and solutions to things I may never get around to programming (or working on a project). Its just how I teach myself, may be a bad method, but thats how I do it. The design that I had going was three seperate servers; account login server, game logic server, and bittorrent file/tracker.

City of Heroes has an instance server which handles the small little "battle" scenes for quests that groups go into. They need this because there are several groups at the same time sometimes doing the same quest, in the same zone, in the same area. They have instances in order to allow people to do the quests at the same time. I think World of Warcraft has something similar (I've been told) but I've never played it.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
Meanwhile, if you want < 2,000 players, you can just stick 'em all on a single server and call it a day ;-)


I agree.. although I would say depending on the problem space the numbers can be somewhat higher. With RunUO the C# Ultima Online emulator it has been tested to have 3200 concurrent TCP connections playing.

Share this post


Link to post
Share on other sites
Sign in to follow this