Other problems will show up long before the servers start chugging, players reciving much more data than their connections can handle would be the main problem. The approach i've taken, with prioritisation of traffic will help but if 1000 players decide to run around in a tiny area then what can you do?
Using agents will have negligable effect on the network, they don't need more bandwidth than a player and a 100mb switched network has a lot of capacity.
An e10k is almost certainly overkill, i'd look at something like a Netra to run multiple zones. For the AI machines where the real computation is i'd pick something like an e420.
The point Kensai raised is interesting, how do you design crowd dispersal without exposing players to underlying zone mechanics? You cant prevent people from crossing zone boundaries.
Kristian
Edited by - Slide on September 10, 2000 1:35:07 AM
MMORPG Theory
Has anyone thought about trying to just evenly distribute the players across servers, and not worrying about where they are geographically? To me, this seems like the embodiment of load balancing, because each time someone logs in, you just figure out which box has the least players and send them there 
Obviously, this would present synchronization (sp?) issues, since players on the same server in all liklihood would be nowhere near each other. So, what I''d suggest is having one server in this cluster (it needn''t be an expensive one either, as I see it) that *does* keep all the players sorted by zone (before I forget, I''m thinking more in terms of a game whose "zones" are transparent to the user, like Asheron''s Call or Ultima Online). Each server knows where the zone bounderies lie, so when a player crosses one it just sends a tiny packet to the "zone server" saying Hey, Anthracks just moved into zone 3.
Now, when a player performs an action that requires others in their zone to know about it, the server sends another little packet to the zone server, saying "send me back an array of all the players in this zone, and the IP of the server they are on". Then, using this list, they forward the action and the list of players on that server that should receive the action to the other servers (presumably almost instantly across a fast LAN connection), and then each server uses that info to send the action down the pipe to clients.
I realize this may put a killer load on the internal network (you''d want a 1Gbit pipe between all the servers probably), but my question is if this method (or some variant) could actually make up for that by making load balancing so much easier to implement. Although you would probably need to make some different server software for that zone server, since its ONLY task is to sort players by zone and send that info back, it would be a relatively tiny bit of code and you could optimize the hell out of it for doing that one task.
Ideas? Comments? Am I psycho? (don''t answer that last one)
Anthracks
Obviously, this would present synchronization (sp?) issues, since players on the same server in all liklihood would be nowhere near each other. So, what I''d suggest is having one server in this cluster (it needn''t be an expensive one either, as I see it) that *does* keep all the players sorted by zone (before I forget, I''m thinking more in terms of a game whose "zones" are transparent to the user, like Asheron''s Call or Ultima Online). Each server knows where the zone bounderies lie, so when a player crosses one it just sends a tiny packet to the "zone server" saying Hey, Anthracks just moved into zone 3.
Now, when a player performs an action that requires others in their zone to know about it, the server sends another little packet to the zone server, saying "send me back an array of all the players in this zone, and the IP of the server they are on". Then, using this list, they forward the action and the list of players on that server that should receive the action to the other servers (presumably almost instantly across a fast LAN connection), and then each server uses that info to send the action down the pipe to clients.
I realize this may put a killer load on the internal network (you''d want a 1Gbit pipe between all the servers probably), but my question is if this method (or some variant) could actually make up for that by making load balancing so much easier to implement. Although you would probably need to make some different server software for that zone server, since its ONLY task is to sort players by zone and send that info back, it would be a relatively tiny bit of code and you could optimize the hell out of it for doing that one task.
Ideas? Comments? Am I psycho? (don''t answer that last one)
Anthracks
I don''t think the model of having x number of people per server instance can scale, as you rightly point out there will be synchronisation issues between each world instance. The problem comes when you start adding more servers, the amount of data being based increases exponentially. 2 servers means 2 updates, 3 servers means 6 updates, 4 servers 12 updates. You get the picture 
The other big problem is of course the world size which will be a limiting factor. You''ll want to have a lot of RAM installed if you want the entire world loaded, and you''ll also need some scheme to assign ownership / responsibility for specific objects in the world.
You can however run seperate worlds in parallel and give the player an option to move his character across servers at will. This of course introduces other issues to do with gameplay & economics.
Kristian
The other big problem is of course the world size which will be a limiting factor. You''ll want to have a lot of RAM installed if you want the entire world loaded, and you''ll also need some scheme to assign ownership / responsibility for specific objects in the world.
You can however run seperate worlds in parallel and give the player an option to move his character across servers at will. This of course introduces other issues to do with gameplay & economics.
Kristian
I see some problems with just distributing the players evenly over the servers and ignoring their geographical position. I believe player interaction, such as combat and fine-grained movement, can produce a lot of traffic so you really want to store players who interact with each other on the same server.
You also need to deal with conflicts between concurrent changes to the world that are happening on different servers. Not only must those changes be replicated between all the servers but the replication must happen very quickly.
Take this situation for example: player A on server 1 pushes a vase off a table, player B on server 2 picks up the vase and player C on server 3 tries to stop the vase from falling off the table, they all do it at the same time.
Handling situations like this will cause a lot of packages to fly between servers and yet the servers must be very quick to agree on what has actually happened in the world since each player expects almost instant response to his action. These situations would increase the complexity of operations and must be handled carefully so they do not cause bugs.
In other words, the model where you ignore the geographical distribution and just care about the player distribution requires an added mechanism to make sure that the world is exactely the same on all the servers at the same time. Sorry if I''m being redundant.
Henry
You also need to deal with conflicts between concurrent changes to the world that are happening on different servers. Not only must those changes be replicated between all the servers but the replication must happen very quickly.
Take this situation for example: player A on server 1 pushes a vase off a table, player B on server 2 picks up the vase and player C on server 3 tries to stop the vase from falling off the table, they all do it at the same time.
Handling situations like this will cause a lot of packages to fly between servers and yet the servers must be very quick to agree on what has actually happened in the world since each player expects almost instant response to his action. These situations would increase the complexity of operations and must be handled carefully so they do not cause bugs.
In other words, the model where you ignore the geographical distribution and just care about the player distribution requires an added mechanism to make sure that the world is exactely the same on all the servers at the same time. Sorry if I''m being redundant.
Henry
Load-balancing should not be much less efficient (from a players-per-server standpoint) if done geographically than by ignoring geography completely. First, assume that every server has a copy of the entire world map and that map is divided into a grid (grid size will take some tuning as I''ll mention later). How many and which grid elements are handled by a server is dependent on server load and how many people are present on each grid element. So Server A may be in charge of 3 grid elements containing 50, 100, and 75 players and Server B might be in charge of 6 grid elements containing 25, 150, 10, 15, 1, and 40 players, respectively. Now as players move around the server architecture could keep track of how many people are where and take charge of various grid elements to balance load. The finer the grid-size the more finely balanced the load but the greater the potential network traffic between servers.
The difficult issue for me has always been how do you handle interactions on grid boundaries? For a small conflict on a boundary the involved servers would have to be in constant communication. For larger or prolonged conflicts, it seems to make sense that the involved grid elements all be handled by a single server. Thus there has to be some method by which the servers can hand entire grid elements off to other servers. Note that with this method the grid elements a given server is handling need not be connected.
This still doesn''t address the fact that player distribution needs to be kept such that no area is so busy as to saturate a player''s 28.8k modem connection. Asheron''s Call has "portal storms" where they basically just teleport players away, based on a FIFO method, but though they make an effort to explain this from a roleplaying perspective it''s pretty apparent to the players what''s going on. Anarchy Online is going to have guards disperse crowds if things get too busy, which seems a bit better, but it remains to be seen how they will handle wilderness areas where it would be difficult explaining the sudden appearance of town guards. Can anyone suggest a better method? I can think of a number of fairly beleivable methods of managing towns, but what''s to stop people from gathering somewhere else? Should it be acceptable to just teleport people out in those situations or to let them lag? I think one of the "holy grails" of MMORPGs is to have massive battles/gatherings but there seems to be no way to get around the limitations of the "weakest link" -- the 28.8k player.
The difficult issue for me has always been how do you handle interactions on grid boundaries? For a small conflict on a boundary the involved servers would have to be in constant communication. For larger or prolonged conflicts, it seems to make sense that the involved grid elements all be handled by a single server. Thus there has to be some method by which the servers can hand entire grid elements off to other servers. Note that with this method the grid elements a given server is handling need not be connected.
This still doesn''t address the fact that player distribution needs to be kept such that no area is so busy as to saturate a player''s 28.8k modem connection. Asheron''s Call has "portal storms" where they basically just teleport players away, based on a FIFO method, but though they make an effort to explain this from a roleplaying perspective it''s pretty apparent to the players what''s going on. Anarchy Online is going to have guards disperse crowds if things get too busy, which seems a bit better, but it remains to be seen how they will handle wilderness areas where it would be difficult explaining the sudden appearance of town guards. Can anyone suggest a better method? I can think of a number of fairly beleivable methods of managing towns, but what''s to stop people from gathering somewhere else? Should it be acceptable to just teleport people out in those situations or to let them lag? I think one of the "holy grails" of MMORPGs is to have massive battles/gatherings but there seems to be no way to get around the limitations of the "weakest link" -- the 28.8k player.
Well some intresting questions there, let me take a stab at some of them 8^)
The concept of multiple servers handling segements of the world, is done by all of the MMORPGs i beleive, so that''s not new. I think its usually implemented by a master database server which stores all the stats of the world, players, monsters, etc.. states which then the mico server (the ones which simulate a small part of the world) queries and feeds data for creating the wolrd simulation. So if the micro servers goes down only the localized data is lost. As for micro server boundaries within the simulation, one could seperate them completely and have no interaction, or have overlaping grid boundaries where control could be shared? Personally I think its more trouble than its worth, just dont have overlaping interactive micro servers.
As for lag, with the increase in bandwidth these days, its proably not bandwidth saturation caused by player clustering which will be the main problem. Mostly likely it will be the simulation performance peanlty for testing player - player collision or some other non-linear computationally intensive task caused by such close quarters player player contact. I''m just guesting here though, it very well could be the bandwidth.
A system using the lock step synchronzation scheme ueses much less bandwidth and the trdational method of predictive/state sychronzation scheme, but suffers from poor performance. If you could implment a hybrid which has the performance of the later and the bandwith usage of the former, then you would have your holy grail of MMORPG 8^)
Good Luck
-ddn
The concept of multiple servers handling segements of the world, is done by all of the MMORPGs i beleive, so that''s not new. I think its usually implemented by a master database server which stores all the stats of the world, players, monsters, etc.. states which then the mico server (the ones which simulate a small part of the world) queries and feeds data for creating the wolrd simulation. So if the micro servers goes down only the localized data is lost. As for micro server boundaries within the simulation, one could seperate them completely and have no interaction, or have overlaping grid boundaries where control could be shared? Personally I think its more trouble than its worth, just dont have overlaping interactive micro servers.
As for lag, with the increase in bandwidth these days, its proably not bandwidth saturation caused by player clustering which will be the main problem. Mostly likely it will be the simulation performance peanlty for testing player - player collision or some other non-linear computationally intensive task caused by such close quarters player player contact. I''m just guesting here though, it very well could be the bandwidth.
A system using the lock step synchronzation scheme ueses much less bandwidth and the trdational method of predictive/state sychronzation scheme, but suffers from poor performance. If you could implment a hybrid which has the performance of the later and the bandwith usage of the former, then you would have your holy grail of MMORPG 8^)
Good Luck
-ddn
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
