#### Archived

This topic is now archived and is closed to further replies.

# MMORPG Theory

This topic is 6337 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hi, this is my first post to this group. Seems like some of you are quite in to this about multiplayer games, so I ask a question: Do anyone know about some written thory about how to make MMORPG''s? I mean the MMORPG-specific themes like bandwidth-utilizing schemes, server-side structure, absolute do''s and dont''s, etc. I have some design and programming experience, so the things about network programming, graphics, etc is covered. Or maybe this theme is to new to have any decent supporting theory yet? Thanks anyway.

##### Share on other sites
I am also researching this area. I have mainly studied server architecture and it seems that the much can be learnt from areas such as database theory, distributed systems and collaborative agents.

Writing a non-collaborative and non-transparent application such as EverQuest should not be too hard (from a server architecture point of view). The real problem arises when you want to create a fully transparent and collaborative distributed server.

Transparent means that the user does not notice server-side operations as opposed to EverQuest where the user clearly notices zone boundaries. Collaborative means that players on different servers can interact, as opposed to EverQuest where you need to be in the same zone to do any real interaction (chat can be sent across zone boundaries).

In other words, the trouble arises when you want to build a distributed zoneless server.

Right now I am looking into good ways to store a dynamic 3d map on a distributed server. Having good memory structures seem really crucial to succeeding with these servers. I have gotten some ideas from multiprocessor operating systems (write-through caches etc.) but I am real interested in hearing what others have come up with.

Henry

##### Share on other sites
One thing important area to research is how to multithread your server.

Games as large as everquest require Solaris as an operating system, NT/2000 won''t cut it, linux might, but Solaris threading model beats any OS out there today.

One important point is the 256 socket per process limitation.
To get around this, you need to create player connection pools that reside in a seperate processes, and have them tunnel their packets to an internal processes on one or more sockets. This way you can have more than 256 players in your game per server, and the processes can reside on another machine if needed to help distribute the load.

This is just a start, servers like these are not easy to create and they take alot of coding/effot and time.

my 2 cents..

##### Share on other sites
I rather like this page...it doesn''t go in-depth on implementation issues, but does provide a nice summary of design-related issues...

http://www.legendmud.org/raph/gaming/index.html

DavidRM
Samu Games

##### Share on other sites
Hi all. As far as I know Everquest uses NT for it's servers, they get around issues with socket limits by not having 256 players in a zone. Not sure if anyone has tested this, but it would be interesting to attempt to get a few hundred players in one of the zones

Implementing a zoneless server isn't much more complex than having a zone based system ala EQ. Origin managed it way back when. Basically, you need to have lots of small zones arranged in a grid fashion.

123456789

The player is in zone 5, therefore they communicate directly with zone 5 and zone 5 is directly responsible for tracking their actions and managing their game. Additionally, zone5 needs to take a read only feed from all the surrounding zones and forward it at a lower priority to the player client. The trick here is to keep the zones small enough so that there isn't more data transfer between the player and servers than the network link could take. The nice thing with this sort of model is that multiple zones could be maintained by one machine.

Voila, one collaborative & distributed world. You'd need to add some sort of master server to service specific requests, do ip:port resolutions for the client & probably handle authentication. There would need to be some serious work involved in preventing stuff like duping and managing objects / players as they switched from zone to zone.

For the network communications, UDP all the way - design the protocol to allow for lossy transmission and lob a timestamp into the packets, and also keep individual packets under a certain size (can't remember correct value, something like 480bytes) so they don't get fragmented.

Ideally you'd have some idea of the bandwidth available on specific connections and configure the verbosity of communications to the client, stuff nearby to the player would take priority - so if someone is lagging you're not sending extraneous information.

I'd be tempted to implement some sort of QOS priority schema on the server also, for every message passed you could prioritise it and stick it in the relevent queue. Priority one probably wouldn't wait until a full packet was available, it would send instantly whereas priority five would wait until all higher priority queues were out of the way. High priority messages would be stuff like a monster hit you, low priority would be someone broadcasting to /ooc or /auction.

One other interesting approach to take, and i'm not sure if this has been tried is to remove the AI logic from the world servers completely. You could run agents that understand the protocols natively on a seperate machine and let them run wild. This would allow for some pretty complex AI, and prevent the world servers from choking to death if the entire population of Orcs wandered into the zone.

Kristian

Edited by - Slide on September 9, 2000 11:50:45 AM

##### Share on other sites
I do not think the zonelessness is as simple as you put it. First you would need the dynamic load-balancing feature that distributes the work that needs to be done over several servers (a mechanism that tries to make sure that each server handles approximately the same amount of players). Each server should also try to handle a separate parts of the geography rather than having several servers handle players that are in the same geographical region.

Complications arise when people change geographical regions often by moving. Changing geographical regions would either cause the player to be moved to a different server or it would cause the server that handles the player to also handle the new region that the player moves to. Other problems would occur if lots of people decide to stand in the same geographical region, more people than can be handled by a single server. It would force several servers to handle the same geographical region and this would cause complications since changes to the region, including movements and actions of all the players in the region, need to be replicated to all the servers that handle players in the region.

Henry

##### Share on other sites
>> As far as I know Everquest uses NT for it''s servers, they get around issues with socket limits by not having 256 players in a zone. >>

Forgot to ask, doesn''t EQ use UDP? In that case, what sockets limits do you have? UDP does not keep a socket open all the time like TCP.

Henry

##### Share on other sites
I don''t see why the work for a specific area needs to be distributed over multiple machines, the issues that are generated in synchronisation are just not worth the trouble. I also don''t think the capacity of the server will be the issue, remember the key resource we are working with here is network capacity not cpu load / memory. It''s comparatively cheap to throw extra memory / more clock cycles at a problem, the real problems will show up on a clients network connection. You cannot send information about 1000 players in a zone down a 28k modem without getting hellish latency. The way I see it, you should take as much of the complexity off the zone server as possible, stuff like creature AI and pathfinding can get chopped pretty easily. This allows you to support many more users on the server than the client could manage.

The biggest problem with multiple small zones as you correctly point out is transferring from one zone to another, this can be mitigated greatly be implementing some kind of caching schema where the zone servers pass client data to adjacent zones in advance of the zone change. There will be some latency during the switch, but with decent predictive modelling that can be minimised.

You were right about EQ though

Kristian

##### Share on other sites
Of course one should do everything to minimize the information that one needs to send to each player. I''ve only been talking about synchronization on the server side, something that would be completely transparent to the player and would be done over an internal 1 Gb network (or whatever the bandwidth the network that the servers are connected through would have).

The thing is, there''s a conflict between distributing the players evenly over the servers and distributing the geography evenly over the servers. You are talking about distributing the geography evenly over the servers by having each server handle a number of determined zones and ignoring the amount of players in each geographical area. This would cause great problems when lots and lots of players move into the same geographical area. Problems that would require some heavy server-side synchronization to solve (or at least that''s the way I see it).

It should be noted that EQ has this problem. Zones often crash when more than 100 players or so move into them at the same time. I believe that finding a good solution to this problem could be quite tricky and that it is not just a matter of reducing the size of the zones.

I do not think you should ignore the hardware issue either. If you want to create a really big MMRPG you will need to have several computers on each server and distribute the load among them. Popping on a few extra megs or an extra processor is easy at first but when you get to the point that you need 4+ processors and 1+ Gb ram you are going into the realm of specialized hardware where the costs rise rapidly. A 32 processor server from Sun could probably run a lot of Eq servers but I doubt it would be cost-efficient since one of those cans costs around \$1000,000 or so.

Henry

##### Share on other sites
With proper load-balancing, I think the limiting factor is going to be a player''s 28.8k modem rather than server load. That is, the server is going to have to find some way to disperse crowds to keep low-bandwidth connections from lagging long before it is unable to handle the client load. A server can certainly handle 100 players, while a 28.8 connection likely can not. This can be mitigated somewhat by taking the UO approach and drastically limiting a player''s awareness of their surroundings (only "hearing" what is said by players onscreen), but it''s still a tough issue.

Another strategy I find intriguing is offloading monster AI & pathfinding onto a separate machine(s) -- the monsters would interact with the servers in much the same was as the rest of the players -- but this might put undue stress on the network...

##### Share on other sites
Other problems will show up long before the servers start chugging, players reciving much more data than their connections can handle would be the main problem. The approach i've taken, with prioritisation of traffic will help but if 1000 players decide to run around in a tiny area then what can you do?

Using agents will have negligable effect on the network, they don't need more bandwidth than a player and a 100mb switched network has a lot of capacity.

An e10k is almost certainly overkill, i'd look at something like a Netra to run multiple zones. For the AI machines where the real computation is i'd pick something like an e420.

The point Kensai raised is interesting, how do you design crowd dispersal without exposing players to underlying zone mechanics? You cant prevent people from crossing zone boundaries.

Kristian

Edited by - Slide on September 10, 2000 1:35:07 AM

##### Share on other sites
Has anyone thought about trying to just evenly distribute the players across servers, and not worrying about where they are geographically? To me, this seems like the embodiment of load balancing, because each time someone logs in, you just figure out which box has the least players and send them there

Obviously, this would present synchronization (sp?) issues, since players on the same server in all liklihood would be nowhere near each other. So, what I''d suggest is having one server in this cluster (it needn''t be an expensive one either, as I see it) that *does* keep all the players sorted by zone (before I forget, I''m thinking more in terms of a game whose "zones" are transparent to the user, like Asheron''s Call or Ultima Online). Each server knows where the zone bounderies lie, so when a player crosses one it just sends a tiny packet to the "zone server" saying Hey, Anthracks just moved into zone 3.

Now, when a player performs an action that requires others in their zone to know about it, the server sends another little packet to the zone server, saying "send me back an array of all the players in this zone, and the IP of the server they are on". Then, using this list, they forward the action and the list of players on that server that should receive the action to the other servers (presumably almost instantly across a fast LAN connection), and then each server uses that info to send the action down the pipe to clients.

I realize this may put a killer load on the internal network (you''d want a 1Gbit pipe between all the servers probably), but my question is if this method (or some variant) could actually make up for that by making load balancing so much easier to implement. Although you would probably need to make some different server software for that zone server, since its ONLY task is to sort players by zone and send that info back, it would be a relatively tiny bit of code and you could optimize the hell out of it for doing that one task.

Anthracks

##### Share on other sites
I don''t think the model of having x number of people per server instance can scale, as you rightly point out there will be synchronisation issues between each world instance. The problem comes when you start adding more servers, the amount of data being based increases exponentially. 2 servers means 2 updates, 3 servers means 6 updates, 4 servers 12 updates. You get the picture

The other big problem is of course the world size which will be a limiting factor. You''ll want to have a lot of RAM installed if you want the entire world loaded, and you''ll also need some scheme to assign ownership / responsibility for specific objects in the world.

You can however run seperate worlds in parallel and give the player an option to move his character across servers at will. This of course introduces other issues to do with gameplay & economics.

Kristian

##### Share on other sites
I see some problems with just distributing the players evenly over the servers and ignoring their geographical position. I believe player interaction, such as combat and fine-grained movement, can produce a lot of traffic so you really want to store players who interact with each other on the same server.

You also need to deal with conflicts between concurrent changes to the world that are happening on different servers. Not only must those changes be replicated between all the servers but the replication must happen very quickly.

Take this situation for example: player A on server 1 pushes a vase off a table, player B on server 2 picks up the vase and player C on server 3 tries to stop the vase from falling off the table, they all do it at the same time.

Handling situations like this will cause a lot of packages to fly between servers and yet the servers must be very quick to agree on what has actually happened in the world since each player expects almost instant response to his action. These situations would increase the complexity of operations and must be handled carefully so they do not cause bugs.

In other words, the model where you ignore the geographical distribution and just care about the player distribution requires an added mechanism to make sure that the world is exactely the same on all the servers at the same time. Sorry if I''m being redundant.

Henry

##### Share on other sites
Load-balancing should not be much less efficient (from a players-per-server standpoint) if done geographically than by ignoring geography completely. First, assume that every server has a copy of the entire world map and that map is divided into a grid (grid size will take some tuning as I''ll mention later). How many and which grid elements are handled by a server is dependent on server load and how many people are present on each grid element. So Server A may be in charge of 3 grid elements containing 50, 100, and 75 players and Server B might be in charge of 6 grid elements containing 25, 150, 10, 15, 1, and 40 players, respectively. Now as players move around the server architecture could keep track of how many people are where and take charge of various grid elements to balance load. The finer the grid-size the more finely balanced the load but the greater the potential network traffic between servers.

The difficult issue for me has always been how do you handle interactions on grid boundaries? For a small conflict on a boundary the involved servers would have to be in constant communication. For larger or prolonged conflicts, it seems to make sense that the involved grid elements all be handled by a single server. Thus there has to be some method by which the servers can hand entire grid elements off to other servers. Note that with this method the grid elements a given server is handling need not be connected.

This still doesn''t address the fact that player distribution needs to be kept such that no area is so busy as to saturate a player''s 28.8k modem connection. Asheron''s Call has "portal storms" where they basically just teleport players away, based on a FIFO method, but though they make an effort to explain this from a roleplaying perspective it''s pretty apparent to the players what''s going on. Anarchy Online is going to have guards disperse crowds if things get too busy, which seems a bit better, but it remains to be seen how they will handle wilderness areas where it would be difficult explaining the sudden appearance of town guards. Can anyone suggest a better method? I can think of a number of fairly beleivable methods of managing towns, but what''s to stop people from gathering somewhere else? Should it be acceptable to just teleport people out in those situations or to let them lag? I think one of the "holy grails" of MMORPGs is to have massive battles/gatherings but there seems to be no way to get around the limitations of the "weakest link" -- the 28.8k player.

##### Share on other sites
Well some intresting questions there, let me take a stab at some of them 8^)

The concept of multiple servers handling segements of the world, is done by all of the MMORPGs i beleive, so that''s not new. I think its usually implemented by a master database server which stores all the stats of the world, players, monsters, etc.. states which then the mico server (the ones which simulate a small part of the world) queries and feeds data for creating the wolrd simulation. So if the micro servers goes down only the localized data is lost. As for micro server boundaries within the simulation, one could seperate them completely and have no interaction, or have overlaping grid boundaries where control could be shared? Personally I think its more trouble than its worth, just dont have overlaping interactive micro servers.

As for lag, with the increase in bandwidth these days, its proably not bandwidth saturation caused by player clustering which will be the main problem. Mostly likely it will be the simulation performance peanlty for testing player - player collision or some other non-linear computationally intensive task caused by such close quarters player player contact. I''m just guesting here though, it very well could be the bandwidth.

A system using the lock step synchronzation scheme ueses much less bandwidth and the trdational method of predictive/state sychronzation scheme, but suffers from poor performance. If you could implment a hybrid which has the performance of the later and the bandwith usage of the former, then you would have your holy grail of MMORPG 8^)

Good Luck

-ddn