Archived

This topic is now archived and is closed to further replies.

Garrland

MMOG Server Architecture

Recommended Posts

Recently I have been scowering the web to find out how the different MMOGs servers are layed out to learn how they work. Here is what seems to be the case with some of the MMOGs. None of this have been verifed with any of the games developers. UO The world is divided into land blocks that are assigned to each server. There seems to be a login server or point of entry. The player is transferred from server to server he/she passes through the land blocks. Everquest The world is really split into zones. Each Server handles 1 of more zones. There also seems to be a login server that acts as a point of entry and track the status of all the zones/servers. The EQ EMU projects source has some really good insight into this. AC2 This game also seems to be split up into land blocks and servers associated with them. What thay call Auth servers are simular to login servers in the previous games mentioned above. AC2 Also seems to have separate "chat" server for normal/allegence/trade communication. All of the games mentioned also have some type of database backend to store information about the world and players. From the dev chats UO developers a few years back some of the terms thay used sounded a lot like Oracle. Does anyone know of any published information about the server architecture of existing MMOGs? [edited by - Garrland on January 17, 2003 2:22:26 PM]

Share this post


Link to post
Share on other sites
Here''s something interesting you missed: http://www.lyrastudios.com/page.php?page=engine_features

The document/architecture links on the right give a decent overview of how the older MMO''s managed their server pools... However, times have changed and most of the new MMO''s don''t go/haven''t gone into details about their server architecture for a few reasons a) it''s a secret (shhh..) b) it''s a hard problem c) not that many people care (surprise!)

However, I can guess how I''d do it...

..(1) || Key: (1-7) - One or more computers with a given function
|| || - Mean Ol'' Firewall
..(2)-||--(4) .. - Fat leased line (gO.C.whatsaffordable)
(3) || -- - Gigabit Ethernet
|| (6) + - Gigabit switch
..(5)-||--+(7)


(1) - A webserver, used to handle patching.. several ways to do this.. client sends its version number, server sends a compressed patch necessary to bring it up to date which the client then applies.. or for a more thorough check the client sends a hash of its files and the server updates them as necessary (and sends any files the client didn''t send hashes for when the client signals it''s finished... so, we have automatic updates. good.

(2) - A login server.. you know the kind.. likely running Win2k with a ton of RAM and multiple(4 or 8) processors, all services shut off except your login application (using I/O Completion ports, natch)

(3) - Chat server.. If you want to have lobby-chat type stuff like EverQuest or Battle.NET you would add another server to handle it as this could add a good deal of strain to your login server if they were the same machine.. implementation is left as an exercise for the programmers machine type depends on how many people you want to be able to chat at once.. this will spike when servers go down obviously.. many MMOs today don''t have chat servers, oh well

(4) - Accounting Database server.. This could be a lot of different things... You''re right in saying several MMOs use Oracle.. however, not all do.. DAoC uses MySQL for it''s database, and I believe the upcoming NCSoft MMO uses MSSQL... I don''t think MMOs put as much strain on database servers as you might think.. This could theoretically run on the same machine as (2) however that just adds extra load and probably isn''t worth it in the long run... this machine should be behind a firewall, secured to the teeth, constantly backed up, with RAID and a few other things to make sure it''s secure and has high data integrity..

(5) - Session server.. at least one, possibly several per ''server'' or ''world'' or ''shard'' or whatever you want to call them.. login server hands the connection over to these guys when the player wants to connect to a world (algorithm likely to include load balancing if there''s multiple session servers per world).. for multiple character servers this would allow you to choose your character, for example and might (I would try to do it this way, at least) handle a player the entire time they are connected to the server/world/shard to avoid slow,unnecessary disconnects/reconnects when switch between type (6) servers which are...

(6) - World/Location servers.. several per server/world/shard.. How these are managed is where the innovation in current MMOs seems to be... basically, the problem is how do you manage handling a lot of players in a small location.. before, if a lot of people gathered in a single area/zone (during a GM event or something) the server handling that area/zone would become incredible overburdened and possibly crash... However it is possible to balance the load amongst multiple servers.. the problem is that having people in close proximity in-game but being managed by different physical servers leads to a certain amount of latency between them and any interactions they might have... There is also added overhead of migrating players from one server to the next (although this shouldn''t be too difficult)..

(7) - Server/World/Shard Database - Manages all the toons/avatars/characters/whatever on a given server/world/shard (why aren''t there a standard names for these things yet?).. should really only be interacted with upon login/period saves/and logout.. like the account database, should have high security and high data integrity...

You could also have some sort of Kerberos like authentication scheme for extra security and for the ability to host server/world/shards from several geographically distant locations with centralized accounting (something like this might have helped Wolfpack, makers of Shadowbane, with their different publishers not being able to play together .. fiasco)

Another possible class of server would be the "server chat server" you mentioned for AC2... Such a server could simply map players with their session servers, be handed incoming chat messages and broadcast them to any players that should recv them.. the chat server would build lists of guilds, groups, etc.. Some advances to this could add in-game mail, message backlogs (for messages missed when away,disconnected, etc), in-game message boards, and a few other neat features many games just don''t implement

Anyway... building an architecture for an MMO is extremely complex and really not as glamourous or catchy as, say, flashy graphics or good game design when it comes to making MMOs... and while there has been a good deal of work done on building highly scalable servers and dynamic load balancing in the business/IT world, the problem becomes more difficult in games as you have to deal with a constant connection and player interaction within a spatial, three-dimensional world (as opposed to serving static and dynamic webpages, although they put more strain on their databases).. Uptime is another problem with current MMOs.. a good server design could reduce downtime for patching, increase uptime during normal operation (by reducing average load, thus increasing MTBF-mean time before failure), and make it possible to handle individual server failure by offloading its work onto other machines... drawing MMO''s ever closer to the 5 9''s (99.999%) uptime that is so loved in the IT world

If you have any thoughts on the subject I''d love to hear them... this stuff interests me as I made the ''mistake'' of becoming a Computer Engineer and not a Computer Scientist

-nohbdy

Share this post


Link to post
Share on other sites
A couple of the more recent (or upcoming) games do not really divide land into zones like that. Citizen Zero (www.citizen-zero.com) uses, as far as I know, a lot of servers that are dynamically assigned "blobs" of land, depending on the server load( ie. how many players are in that area), so as people move around, the servers switch around to cover the areas where people actually are. The whole world is also (from a players viewpoint) one big zone, with no loading when going from one part to another, or from outdoors to indoors areas.

I can''t remember any more details on how it worked, but if you do a search on "BigWorld" (The game engine they use), you should find a couple of articles about it.

Share this post


Link to post
Share on other sites
If you’re really interested, Bigworld''s site has some pretty interesting information on how their server-client system works.

Bigworld Technology assigns a cluster of machines (game servers) to run each game "world" and then assigns each server to a different area of the world to manage all the players, essentially breaking the world up into different areas of control.


Areas of Control:

Areas of control are assigned dynamically and can be changed at a moment’s notice (without any noticeable change in gameplay quality, most of the time). More than one server can be assigned to one area of the game world, and the number of servers handling a specific ‘Game Area’ changes as the load increases/decreases.

Player Management:

Player Management is also handled dynamically (DUH!) and (depending on the player’s latency, location, etc.) is assigned to be handled by one of the Game Servers controlling the Game Area in question.

In general both Areas of Control and Player Management are assigned ‘ on the fly’ (dynamically), and are managed by negotiation. The entire cluster of servers balances the load between all the servers at constant intervals (The site claims 20 ms). If a server (or an area of servers controlling one location) is being pushed beyond its means and risks failure or increased ‘lag’, then the cluster effectively balances the load to handle the increased demand. (Load balancing/management)


The covers some of the basics of the BigWorld specific technology. There is more information at their site


On a sidenote, I would like to agree with whoever mentioned that there should be some sort of standard with the terminology used. It can get a wee bit confusing sometimes. :/


My 2 cents.

Cheers,

Raymondo.

Share this post


Link to post
Share on other sites
Another good game to look at is EQoA, the new PS2 game coming out. They have completely seamless zones, including dungeons. It''s quite nice. They are planning to support 5000+ per ''world'' server.

Keep in mind that it''s real easy with these games to over complicate the design. Remember the KISS acronym? Many in the latest batch of MMO programmers (hobbyists and pros alike) seem to have forgotten the ''keep it simple stupid'' part of engineering. Having a cool back-end design full of techno-babble terms doesn''t necessarily net you more players (i.e. customers).

Share this post


Link to post
Share on other sites
A good idea in recent(ish) years is to have your server side object very fluid and able to transfer control to another server machine. This can be done by a request from the object or any valid server.
Typically, but not always, server machines are close together in terms of network topography so the user will not notice when an object does such transfers.
This is also good practise for fault recovery since if one server machine goes down it is entirely possible to have all the important objects transfer over to a different machine.


Martin Piper
Argonaut Games plc.

Share this post


Link to post
Share on other sites
This is all great input from everyone. I just read the www.lyrastudios.com site mentioned above and I has some good information. Also, after I posted the topic I received my latest issue of Game Developer and they had a whole article about Zona's Terazona and another company that was not BigWorld. I'm at work so I can't look it up. Very informative article but both used MMOG architecture close to the way Lyra Studio's works with 1 and only 1 server assigned to parts of the world. BigWorld from what I have been able to find out is the only MMOG architecture that has multiple servers handle the load in the same location by dynamically load balancing a pool of servers.

By having multiple servers share the load for a concentrated area in the game world, wouldn't you still have issues with overloading the client with information? Sure servers can handle the load but can the players handle the CPU/Bandwidth requirements for displaying/receiving data for that many players in the same area?

To help CPU you could use LOD and degenerate the models/terrain to handle that many players on the screen. I have seen MMOGs use this for general rendering but not scale by number of players visible on the screen. I have seen some single player 3D RTSs use the technique.

For bandwidth, you could only send information for players in your view. This could be limited by your game design if you had a radar like AC2 you would still need information about players behind you but not as much if in front of you. This would put more load on the server but using BigWorld's system this would be scalable. You could also Prioritize game data and reduce the amount of detail of player activities the farther away other players are from view.

I know I expanded this topic a beyond the scope of the original post but everyone providing great input.



[edited by - Garrland on January 22, 2003 1:51:20 PM]

Share this post


Link to post
Share on other sites
Yeah, the client can get "overloaded" if too many people are in the same surrounding area, and that can be a problem... But like you said there are several things both the client and server can do to speed it up.. on the rendering side, the client has good culling so it doesn''t waste time on models it can''t see and a good system for dynamic LOD (though it seems many games today are fill-rate limited so throwing more polygons at modern video cards doesn''t hurt as much as long as there''s no overdraw).. also there''s of course the user-settable options like whether or not to showcorpses, whether or not to show particle effects, certain effects like shaders can be disabled, etc etc...

the server side might add the ability to prioritize updates so the objects the player is actually interacting with are updated more frequently (not necessarily the ones that are closest or in view, think range combat or long range magic, though that would certainly factor into it) and would have some tricks to make updates smaller (eg partial updates, compression) to avoid wasting bandwidth and user settable options like Everquest has to turn off data that the player may or may not want to get sent to them.. like battlespam...

to be honest though, the main reason companies try to save bandwidth is so they can keep their costs down.. bandwidth is expensive for MMOs and anything you can do to minimize it is doubleplus good... if it helps the poor modem users game run better in a crowded area or large fight, all the better

Share this post


Link to post
Share on other sites
I guess one thing you could do is reduce the network update time of objects that are far away from the player. That way the client receives more frequent updates of objects that are close by, while ones further away might be update say once per second.

Just a thought, and by no means backed by experience.

Share this post


Link to post
Share on other sites
All right, so we all pretty-much understand the basic architecture that most of the modern MMOG systems are using...

So what about an online RPG/FPS? How on earth are servers going to be able to handle all that information, and update it in a quickly enough?

One specific game that comes to mind when I think about an RPG/FPS is the upcoming PlanetSide


Cheers,

Raymondo


[edited by - raymondo on January 26, 2003 6:46:50 PM]

Share this post


Link to post
Share on other sites
raymondo - keep in mind that very few MMO projects hit their shipping date So you aren''t the only one wondering how to get this done right...

Planetside looks pretty cool IMO, but yeah, they will have some lag issues I''m guessing. I''d imagine they''ll use UDP to stream the data, and use client-side prediction to fill the gaps. Pretty standard stuff, but it works well for FPS games. Planetside isn''t just a MMO FPS though, looks like they''ll have some other cool gameplay features (along with persistence).

As far as updating the client in a RTS goes, I think the key here is entity density. If you have a huge game world, then you are likely to have only as many units in an area as you would in a game like WarCraft3, or StarCraft (I''m a blizzard fan, sorry). Also, notice you control fewer units in WC3 I''m currently working on a project that is MMORTS. I use a small number of units per player, in conjunction with huge areas to minimize the entity density of a given update region.

Share this post


Link to post
Share on other sites