MMORPG Theory
Author
Hi, this is my first post to this group. Seems like some of you are quite in to this about multiplayer games, so I ask a question: Do anyone know about some written thory about how to make MMORPG''s? I mean the MMORPG-specific themes like bandwidth-utilizing schemes, server-side structure, absolute do''s and dont''s, etc. I have some design and programming experience, so the things about network programming, graphics, etc is covered. Or maybe this theme is to new to have any decent supporting theory yet?
Thanks anyway.
I am also researching this area. I have mainly studied server architecture and it seems that the much can be learnt from areas such as database theory, distributed systems and collaborative agents.
Writing a non-collaborative and non-transparent application such as EverQuest should not be too hard (from a server architecture point of view). The real problem arises when you want to create a fully transparent and collaborative distributed server.
Transparent means that the user does not notice server-side operations as opposed to EverQuest where the user clearly notices zone boundaries. Collaborative means that players on different servers can interact, as opposed to EverQuest where you need to be in the same zone to do any real interaction (chat can be sent across zone boundaries).
In other words, the trouble arises when you want to build a distributed zoneless server.
Right now I am looking into good ways to store a dynamic 3d map on a distributed server. Having good memory structures seem really crucial to succeeding with these servers. I have gotten some ideas from multiprocessor operating systems (write-through caches etc.) but I am real interested in hearing what others have come up with.
Henry
Writing a non-collaborative and non-transparent application such as EverQuest should not be too hard (from a server architecture point of view). The real problem arises when you want to create a fully transparent and collaborative distributed server.
Transparent means that the user does not notice server-side operations as opposed to EverQuest where the user clearly notices zone boundaries. Collaborative means that players on different servers can interact, as opposed to EverQuest where you need to be in the same zone to do any real interaction (chat can be sent across zone boundaries).
In other words, the trouble arises when you want to build a distributed zoneless server.
Right now I am looking into good ways to store a dynamic 3d map on a distributed server. Having good memory structures seem really crucial to succeeding with these servers. I have gotten some ideas from multiprocessor operating systems (write-through caches etc.) but I am real interested in hearing what others have come up with.
Henry
One thing important area to research is how to multithread your server.
Games as large as everquest require Solaris as an operating system, NT/2000 won''t cut it, linux might, but Solaris threading model beats any OS out there today.
One important point is the 256 socket per process limitation.
To get around this, you need to create player connection pools that reside in a seperate processes, and have them tunnel their packets to an internal processes on one or more sockets. This way you can have more than 256 players in your game per server, and the processes can reside on another machine if needed to help distribute the load.
This is just a start, servers like these are not easy to create and they take alot of coding/effot and time.
my 2 cents..
Games as large as everquest require Solaris as an operating system, NT/2000 won''t cut it, linux might, but Solaris threading model beats any OS out there today.
One important point is the 256 socket per process limitation.
To get around this, you need to create player connection pools that reside in a seperate processes, and have them tunnel their packets to an internal processes on one or more sockets. This way you can have more than 256 players in your game per server, and the processes can reside on another machine if needed to help distribute the load.
This is just a start, servers like these are not easy to create and they take alot of coding/effot and time.
my 2 cents..
I rather like this page...it doesn''t go in-depth on implementation issues, but does provide a nice summary of design-related issues...
http://www.legendmud.org/raph/gaming/index.html
DavidRM
Samu Games
http://www.legendmud.org/raph/gaming/index.html
DavidRM
Samu Games
Hi all. As far as I know Everquest uses NT for it's servers, they get around issues with socket limits by not having 256 players in a zone. Not sure if anyone has tested this, but it would be interesting to attempt to get a few hundred players in one of the zones 
Implementing a zoneless server isn't much more complex than having a zone based system ala EQ. Origin managed it way back when. Basically, you need to have lots of small zones arranged in a grid fashion.
The player is in zone 5, therefore they communicate directly with zone 5 and zone 5 is directly responsible for tracking their actions and managing their game. Additionally, zone5 needs to take a read only feed from all the surrounding zones and forward it at a lower priority to the player client. The trick here is to keep the zones small enough so that there isn't more data transfer between the player and servers than the network link could take. The nice thing with this sort of model is that multiple zones could be maintained by one machine.
Voila, one collaborative & distributed world. You'd need to add some sort of master server to service specific requests, do ip:port resolutions for the client & probably handle authentication. There would need to be some serious work involved in preventing stuff like duping and managing objects / players as they switched from zone to zone.
For the network communications, UDP all the way - design the protocol to allow for lossy transmission and lob a timestamp into the packets, and also keep individual packets under a certain size (can't remember correct value, something like 480bytes) so they don't get fragmented.
Ideally you'd have some idea of the bandwidth available on specific connections and configure the verbosity of communications to the client, stuff nearby to the player would take priority - so if someone is lagging you're not sending extraneous information.
I'd be tempted to implement some sort of QOS priority schema on the server also, for every message passed you could prioritise it and stick it in the relevent queue. Priority one probably wouldn't wait until a full packet was available, it would send instantly whereas priority five would wait until all higher priority queues were out of the way. High priority messages would be stuff like a monster hit you, low priority would be someone broadcasting to /ooc or /auction.
One other interesting approach to take, and i'm not sure if this has been tried is to remove the AI logic from the world servers completely. You could run agents that understand the protocols natively on a seperate machine and let them run wild. This would allow for some pretty complex AI, and prevent the world servers from choking to death if the entire population of Orcs wandered into the zone.
Kristian
Edited by - Slide on September 9, 2000 11:50:45 AM
Implementing a zoneless server isn't much more complex than having a zone based system ala EQ. Origin managed it way back when. Basically, you need to have lots of small zones arranged in a grid fashion.
123456789 The player is in zone 5, therefore they communicate directly with zone 5 and zone 5 is directly responsible for tracking their actions and managing their game. Additionally, zone5 needs to take a read only feed from all the surrounding zones and forward it at a lower priority to the player client. The trick here is to keep the zones small enough so that there isn't more data transfer between the player and servers than the network link could take. The nice thing with this sort of model is that multiple zones could be maintained by one machine.
Voila, one collaborative & distributed world. You'd need to add some sort of master server to service specific requests, do ip:port resolutions for the client & probably handle authentication. There would need to be some serious work involved in preventing stuff like duping and managing objects / players as they switched from zone to zone.
For the network communications, UDP all the way - design the protocol to allow for lossy transmission and lob a timestamp into the packets, and also keep individual packets under a certain size (can't remember correct value, something like 480bytes) so they don't get fragmented.
Ideally you'd have some idea of the bandwidth available on specific connections and configure the verbosity of communications to the client, stuff nearby to the player would take priority - so if someone is lagging you're not sending extraneous information.
I'd be tempted to implement some sort of QOS priority schema on the server also, for every message passed you could prioritise it and stick it in the relevent queue. Priority one probably wouldn't wait until a full packet was available, it would send instantly whereas priority five would wait until all higher priority queues were out of the way. High priority messages would be stuff like a monster hit you, low priority would be someone broadcasting to /ooc or /auction.
One other interesting approach to take, and i'm not sure if this has been tried is to remove the AI logic from the world servers completely. You could run agents that understand the protocols natively on a seperate machine and let them run wild. This would allow for some pretty complex AI, and prevent the world servers from choking to death if the entire population of Orcs wandered into the zone.
Kristian
Edited by - Slide on September 9, 2000 11:50:45 AM
I do not think the zonelessness is as simple as you put it. First you would need the dynamic load-balancing feature that distributes the work that needs to be done over several servers (a mechanism that tries to make sure that each server handles approximately the same amount of players). Each server should also try to handle a separate parts of the geography rather than having several servers handle players that are in the same geographical region.
Complications arise when people change geographical regions often by moving. Changing geographical regions would either cause the player to be moved to a different server or it would cause the server that handles the player to also handle the new region that the player moves to. Other problems would occur if lots of people decide to stand in the same geographical region, more people than can be handled by a single server. It would force several servers to handle the same geographical region and this would cause complications since changes to the region, including movements and actions of all the players in the region, need to be replicated to all the servers that handle players in the region.
Henry
Complications arise when people change geographical regions often by moving. Changing geographical regions would either cause the player to be moved to a different server or it would cause the server that handles the player to also handle the new region that the player moves to. Other problems would occur if lots of people decide to stand in the same geographical region, more people than can be handled by a single server. It would force several servers to handle the same geographical region and this would cause complications since changes to the region, including movements and actions of all the players in the region, need to be replicated to all the servers that handle players in the region.
Henry
>> As far as I know Everquest uses NT for it''s servers, they get around issues with socket limits by not having 256 players in a zone. >>
Forgot to ask, doesn''t EQ use UDP? In that case, what sockets limits do you have? UDP does not keep a socket open all the time like TCP.
Henry
Forgot to ask, doesn''t EQ use UDP? In that case, what sockets limits do you have? UDP does not keep a socket open all the time like TCP.
Henry
I don''t see why the work for a specific area needs to be distributed over multiple machines, the issues that are generated in synchronisation are just not worth the trouble. I also don''t think the capacity of the server will be the issue, remember the key resource we are working with here is network capacity not cpu load / memory. It''s comparatively cheap to throw extra memory / more clock cycles at a problem, the real problems will show up on a clients network connection. You cannot send information about 1000 players in a zone down a 28k modem without getting hellish latency. The way I see it, you should take as much of the complexity off the zone server as possible, stuff like creature AI and pathfinding can get chopped pretty easily. This allows you to support many more users on the server than the client could manage.
The biggest problem with multiple small zones as you correctly point out is transferring from one zone to another, this can be mitigated greatly be implementing some kind of caching schema where the zone servers pass client data to adjacent zones in advance of the zone change. There will be some latency during the switch, but with decent predictive modelling that can be minimised.
You were right about EQ though
Kristian
The biggest problem with multiple small zones as you correctly point out is transferring from one zone to another, this can be mitigated greatly be implementing some kind of caching schema where the zone servers pass client data to adjacent zones in advance of the zone change. There will be some latency during the switch, but with decent predictive modelling that can be minimised.
You were right about EQ though
Kristian
Of course one should do everything to minimize the information that one needs to send to each player. I''ve only been talking about synchronization on the server side, something that would be completely transparent to the player and would be done over an internal 1 Gb network (or whatever the bandwidth the network that the servers are connected through would have).
The thing is, there''s a conflict between distributing the players evenly over the servers and distributing the geography evenly over the servers. You are talking about distributing the geography evenly over the servers by having each server handle a number of determined zones and ignoring the amount of players in each geographical area. This would cause great problems when lots and lots of players move into the same geographical area. Problems that would require some heavy server-side synchronization to solve (or at least that''s the way I see it).
It should be noted that EQ has this problem. Zones often crash when more than 100 players or so move into them at the same time. I believe that finding a good solution to this problem could be quite tricky and that it is not just a matter of reducing the size of the zones.
I do not think you should ignore the hardware issue either. If you want to create a really big MMRPG you will need to have several computers on each server and distribute the load among them. Popping on a few extra megs or an extra processor is easy at first but when you get to the point that you need 4+ processors and 1+ Gb ram you are going into the realm of specialized hardware where the costs rise rapidly. A 32 processor server from Sun could probably run a lot of Eq servers but I doubt it would be cost-efficient since one of those cans costs around $1000,000 or so.
Henry
The thing is, there''s a conflict between distributing the players evenly over the servers and distributing the geography evenly over the servers. You are talking about distributing the geography evenly over the servers by having each server handle a number of determined zones and ignoring the amount of players in each geographical area. This would cause great problems when lots and lots of players move into the same geographical area. Problems that would require some heavy server-side synchronization to solve (or at least that''s the way I see it).
It should be noted that EQ has this problem. Zones often crash when more than 100 players or so move into them at the same time. I believe that finding a good solution to this problem could be quite tricky and that it is not just a matter of reducing the size of the zones.
I do not think you should ignore the hardware issue either. If you want to create a really big MMRPG you will need to have several computers on each server and distribute the load among them. Popping on a few extra megs or an extra processor is easy at first but when you get to the point that you need 4+ processors and 1+ Gb ram you are going into the realm of specialized hardware where the costs rise rapidly. A 32 processor server from Sun could probably run a lot of Eq servers but I doubt it would be cost-efficient since one of those cans costs around $1000,000 or so.
Henry
With proper load-balancing, I think the limiting factor is going to be a player''s 28.8k modem rather than server load. That is, the server is going to have to find some way to disperse crowds to keep low-bandwidth connections from lagging long before it is unable to handle the client load. A server can certainly handle 100 players, while a 28.8 connection likely can not. This can be mitigated somewhat by taking the UO approach and drastically limiting a player''s awareness of their surroundings (only "hearing" what is said by players onscreen), but it''s still a tough issue.
Another strategy I find intriguing is offloading monster AI & pathfinding onto a separate machine(s) -- the monsters would interact with the servers in much the same was as the rest of the players -- but this might put undue stress on the network...
Another strategy I find intriguing is offloading monster AI & pathfinding onto a separate machine(s) -- the monsters would interact with the servers in much the same was as the rest of the players -- but this might put undue stress on the network...
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
