Sign in to follow this  
Antheus

MMOG, message passing, proxies and clients

Recommended Posts

I'm looking for a little insight on viability of a clustered MMOG server design with respect to different physical server layouts. Server logic is performed by message passing. Central directory contains a list of topics, any message can be passed to any topic. Each process interested into receiving messages has access to a daemon that maintains local snapshot of topics, updates remote subscriptions and publishes local subscription changes. Messages are passed between processes in a peer-to-peer fashion. Broadcasting of messages to interested subscribers within the process is performed by the daemon. This allows for minimal ammount of traffic within cluster network. World consists out of zones, each of which is distributed across several physical machines. Each zone publishes events via messages to a single topic. In addition, some messages need to be passed directly to individual client. Clients connections are handled by proxy servers. Proxy servers are also responsible for broadcasting global messages (changes in the zone) to individual clients. For example: When an object moves, individual zone server publishes message to the global zone topic, this message is propagated once to each proxy server, proxy server then performs actual broadcast to clients in range. One issue I'm having some considerations about is how individual clients communicate with the world. Current design intends to create two topics for each individual client. When a connection to the client is established, proxy server registers two new topics, one to which the client publishes, and a new topic to which any world server may publish. The number of clients is assumed to be in 4 digit range. Example of topics in such world would be: - WorldZone1 - Chat - Client1In - Client1Out - Client2In - Client2Out From design perspective, this seems rather clean and simple. When client object moves through the world, individual servers simply subscribe and unsubscribe to topics for the objects they handle. The total number of topics does not apear to be an issue here, and topic and topic subscription management overhead seems to be minimal (client sessions last for at least several minutes and up to several hours). Due to central subscription management, all physical servers can pass messages to any subscriber, but are still guaranteed that cluster network load will be optimal, as individual daemons pass the physical messages only to interested parties without flooding every daemon on the network. An additional benefit here is that since each client has individual topic (rather than common topic like movement or combat), messages from clients are delivered only to a subset of physical servers, rather than to every server that handles this type of messages. The following are given: - Anyone can publish any message to any topic. Subscribers will handle unknown messages apropriately - Messaging system will ensure that messages published to a topic are delivered to all subscribers, and only via daemons that actually have subscribers for specific topic - Messages may be queued (in bounded or unbounded fashion) by daemon to handle congestions - All participating servers are capable of accepting/publishing messages up to physical network capacity (network is the bottleneck). Logic of handling the actions performed by messages is not included here, just the transport load. - If publishers/subscribers are located in the same process, messages are passed directly within same thread without serialization (threading/locking issues are considered here). Transparency is ensured by the daemon. - Load estimates were done for 2000 users, each client sends an average of 5 messages per second, each proxy server holds 200 users (or whichever number will be convenient), messages are smaller than minimum MTU (<500 bytes). - Clients cannot be migrated between proxy servers without disconnecting first. The question here apply to how logic will perform (with regards to design of messaging system and world/client topics): - Is this reasonable, or are the any warning signs that pop up to someone who has done similar systems? - Is topic-per-client aproach too generic, too hard to administer, too resource intensive? - Assuming the requirement that messages can be broadcast in arbitrary fashion (global events, one-client-only, multiple-clients), is there an alternative that is as close to optimal as possible? - Are there any practical, or even design issues that I'm missing? - Are there any unexpected bottlenecks?

Share this post


Link to post
Share on other sites
Routing through topic look-up for each message sounds heavy-weight; it'll likely be a scalability bottleneck.

Also, you should consider how a world server knows which client topics to subscribe to, and how a client knows which world topics to subscribe to, and what happens when the client moves around in the world.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
Routing through topic look-up for each message sounds heavy-weight; it'll likely be a scalability bottleneck.


Topic look-ups are local, and performed only at start of process, or when a client connects/disconnects. So one local hash-map look-up per lifetime of publisher. List of available topics is broadcast by topic directory, and each client maintains a up-to-date local list.

Unmarshalling a message is once again just a hash map lookup. Given that the serialization is demanding by itself, one hash map lookup seems negligible. Local broadcast (inside proxy) has a bottleneck in multicasting the message (up to several hundred clients), but it only requires one look-up (resolving marshalled topic id into local subscriber list).

Is there an alternative? While many MMOG designs use messaging schemes, I've found them to require excessive local bandwidth, or use very complex node migration schemes which aren't compatible with my requirements.

Quote:
Also, you should consider how a world server knows which client topics to subscribe to, and how a client knows which world topics to subscribe to, and what happens when the client moves around in the world.


Each client is represented in the world with an object. Topic name of a client is derived directly from that particular object's ID.

Client connection (actual player) is always connected to one single proxy. As client object(s) move through different zones, or between servers in same zone, they are marshalled, and zone server into which they are inserted connects to associated topic, whereas previous server unsubscribes.

This is the reason why I find such design apealing - it completely separates player's world location from physical client connection. In my case, migrating client connection between zones/shards isn't an option.

Share this post


Link to post
Share on other sites
So you'd have a separate channel where the proxy server can tell a given world to create/demarshal a new player object (and, later, to remove, when player changes zone).

I see two possible problems up front:
1) A race condition between the player object being made available to the zone server, and the central authority broadcasting the new client channel availability. When the player object comes to the zone, the new channel may not yet be available. There are various ways to solve this.
2) Local bandwidth on the network between the proxies and the zones. Depending on the subscription model, this may be saturated. I would assume that, in practice, each proxy server would be subscribed to each world server, as they'd have some player from each zone (statistically). Thus, every zone sends to every proxy.

Also, the central channel broadcast packet would become big, with a thousand players (2000 channels for that), some dozens of zones, and then channels for things like chat, NPCs, guilds, and other things, you're looking at many kilobytes -- broadcasts using Ethernet fragmentation might be a bad idea.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
So you'd have a separate channel where the proxy server can tell a given world to create/demarshal a new player object (and, later, to remove, when player changes zone).

I see two possible problems up front:
1) A race condition between the player object being made available to the zone server, and the central authority broadcasting the new client channel availability. When the player object comes to the zone, the new channel may not yet be available. There are various ways to solve this.


When a client connects, the two related topics are created. These persist for the duration of client's connection. So if a topic is destroyed during zone migration, and new zone cannot find it, it many persist player objects back into database, since client no longer exists.

Temporary latency of topic creation are handled by implementation itself, which buffers outgoing messages for given topic until directory updates the state.

Quote:

2) Local bandwidth on the network between the proxies and the zones. Depending on the subscription model, this may be saturated. I would assume that, in practice, each proxy server would be subscribed to each world server, as they'd have some player from each zone (statistically). Thus, every zone sends to every proxy.


This is true. Unfortunately, the world design is somewhat opaque with all world related information relying on spatial data, so it cannot be partitioned much further. In addition, all world data is dynamic, so it cannot be pre-loaded in form of static models, or pre-calculated collision maps. Entire world consists out of up to 6 million objects, so partitioning the world by different interaction types isn't viable, since even limited object ghosting demands too much memory or would require huge on-demand object marshalling between various subsystems.

Main solution to this problem is handled by the subscription system on object level, which calculates area of interest for individual players. When such client subscribers, objects within that AoI are marshalled to the client, but proxy is also maintains the reference count (how many of its clients track certain object). Messages relating to changes of world objects will therefore be sent only proxies that can forward them to clients, regardless of their topic subscription. This filtering is done at higher level.

The AoI subscriptions are handled periodically, if object has moved a sufficient ammount.

There are two cases to this.
Players will, for most part, remain somewhat grouped. Even in worst case scenario, if all clients on a single proxy do not share AoI, and are all in different zones, they still represent only a small subset of all world objects.

Game systems that do use ghosted objects will most likely be either mapped directly to individual zone servers, or contain very limited subset of object types.

The main demand turns out not to be so much the logic, but more the sheer number of different (and distinct) inter-related world objects which cannot be simplified or templated. So just splitting physical locations across different servers solves most of the problems. Unfortunately, as mentioned, spatial relations are vital, so they prevent use of DHT or similar partitioning, as they would either require too much memory, or result in too many remote calls.

Each client is assumed to know about 10,000 objects at most, although typical number should be around 3000.

Quote:

Also, the central channel broadcast packet would become big, with a thousand players (2000 channels for that), some dozens of zones, and then channels for things like chat, NPCs, guilds, and other things, you're looking at many kilobytes -- broadcasts using Ethernet fragmentation might be a bad idea.


Broadcast here is the for loop that sends same message to all receivers (less or equal to number of cluster nodes). All topics with exception of clients are assumed to exist during the lifetime of the server. While a publisher (individual zone server/chat server) may go down in between, the subscribers continue listening until it's back up again.

Client topics exist for the duration of client connection. After client disconnects, it can no longer reconnect to existing session, and must receive world state from scratch. So if a required client specific topic doesn't exist, all client related info is unloaded from the world.

Share this post


Link to post
Share on other sites
Quote:
they would either require too much memory, or result in too many remote calls


You might want to check that assumption :-)

We spread our simulation geographically, on simulation servers. Then we have communication servers, which are tied to the simulation servers. The players connect to communication servers in their AoI. That way, the communication servers don't need to connect to spatially disperse players, and the N-squared communication is reduced to N.

For interactions, any object is guaranteed to see an instance of any other object in its sphere of interest; either the authoritative copy, or a ghost copy. There are no "remote calls" in our system (I'm not a fan of RPC for real-time use); instead, there is just messaging. We have to accept a latency of one message phase for interactions between two actor (player) objects; other objects can determine the interaction response using deterministic co-simulation.

More information on our system/platform, if you're interested.

Share this post


Link to post
Share on other sites
Unfortunately until you run tests with actual loads you cant always predict where or when it will bottleneck. Random local increases and common patterns in player/object densities (the N^2 effect and the bank phenomenon) can stress the system and bring major degredation.

You need to not only run the message passing loads but also reasonable game mechanics logic and script AI to cause a more realistic exercise of the system.
(cache effects by larger data set access patterns can greatly lower performance measured without a full load).


If peer servers process within a zone (scaleability by load leveling) and one peer bottlenecks will not events that send replies/post status generally cascade degredation to the other peers???

Ordinary objects within the zone are duplicated on each of the peer servers??
(thus requiring propagating updates for all interactions and conflict resolution if more than one action on different peer servers happen simultaneously to the same object -- unified resolution of action effects is needed for a 'turn').



I assume this is a bubble world type zone boundry system and not transparent boundries (which requires 'uglier' multiple place holder proxies and zone transfer complications and closer timing).

[Edited by - wodinoneeye on March 7, 2007 10:17:06 PM]

Share this post


Link to post
Share on other sites
Again with the assumptions :-)

Our system has been in production for five years, although we've now significantly improved it since those early days. It is not a "bubble world," it is a transparently zoned world, that serves worlds up to the size of whole-Earth. Applications range from entertainment (MTV Virtual Laguna Beach; There.com) through medical, transportation, homeland security and military. As far as I know, it is the only successfully applied MMO technology for "serious games."

Anyway, feel free to implement your system and report back on the findings under load. I'll be very interested!

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
Again with the assumptions :-)

Our system has been in production for five years, although we've now significantly improved it since those early days. It is not a "bubble world," it is a transparently zoned world, that serves worlds up to the size of whole-Earth. Applications range from entertainment (MTV Virtual Laguna Beach; There.com) through medical, transportation, homeland security and military. As far as I know, it is the only successfully applied MMO technology for "serious games."

Anyway, feel free to implement your system and report back on the findings under load. I'll be very interested!




Are you using many small 'zones' or larger zones ?? (small for example would be the ones in my project which the players 'view' is a 15x15 window of them versus something large like UOs where most players are on average not near a boundry..)

The simulation that Im working towards (also with transparent zone transitions) has a high density of local detail and a low user count so I can employ a sparse in-memory system (with abstracted world interactions for intelligent 'significant' entities and ecological/economic systems -- and 'realizing' them to full detail only when the player is close by).


Im not sure how well the OPs system (if I interpretted it right) would work having a zone run on several peer servers. My system has each zone running on one ZoneServer that handles the interaction bookkeeping and proxies, and there are seperate AInode(s) that the AI for NPCs run on. More than one zone can run per ZoneServer (load leveled), but each zone's processing has one centralized location.
The multiple proxy complication at the zone boundries is not too bad as the AINode entities/Player clients are already remote (handoff is lightweight as they are little more than listener registrations).


Share this post


Link to post
Share on other sites
The size and shape of our zones is set by the operations staff, and can vary from one 100,000,000 square kilometers down to 25x25 meters or so. (Could be smaller, but then the quad tree starts becoming cumbersomely large :-)

If you have sparse players, Bob's your uncle -- you only need to instantiate (or page in) the parts where players actually are. Also, spatial culling will work wonders, giving collision tests only against a small subset of your world in any one step.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
The size and shape of our zones is set by the operations staff, and can vary from one 100,000,000 square kilometers down to 25x25 meters or so. (Could be smaller, but then the quad tree starts becoming cumbersomely large :-)

If you have sparse players, Bob's your uncle -- you only need to instantiate (or page in) the parts where players actually are. Also, spatial culling will work wonders, giving collision tests only against a small subset of your world in any one step.




Those distances arent significant unless I then knew what the NPCs/Players sensor distances are (ie- 1km range with the 25x25m means alot of zones to subscribe to...). I would expect that you use smaller zones for interiors and extra dense areas of activity, but do you allow variable zone dimensions (and possibly irregular sections, tho still Axis Aligned..) or simple rectangles.


AAAAAAAAAAA
AAAAAABBBAA
AAAAAABBBAA ... B being a high content 'detail' zone
AAAAAABBBAA ... surrounded by an mostly empty A zone
AAAAAAAAAAA
AAAAAAAAAAA
AAAAAAAAAAA



I can use a sparse system (with rollout and creation/expansion on the fly) and even within the zones there is partitioning (I so far have a budget of 2000 'dumb' objects each). Each zone is a fixed data structure (65k at this point) with a local heap to simplify the rollin/out. Intelligent 'significant' objects
live in their own data space (and can rollin and out if needed).

There are actually 3 types of 'zone' -- one being a regular heightmap terrain 'chunk', another being 'structures' (an irregular navmesh) that sits ontop of the heightmap terrain which is also used for vehicles (and I suppose for terrain the heightmap cant handle) and a 'tunnel' type used for interiors (also a navmesh based).



Share this post


Link to post
Share on other sites
Just to clarify this. The system design (client cannot change connections to cluster mid-session) is given and something that cannot be changed.

In addition, the clients are unaware of any kind of clustering, so this aspects needs to be completely transparent. They also assume that any zone is available at any time and they can switch between zones freely and instantly without restrictions. The only physical separation that exists is between individual zones, but the transitions between cluster nodes within individual zone must be transparent.

So this is not about designing the latest greates MMO server. It's just a technical issue of retro-fitting an existing system.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Just to clarify this. The system design (client cannot change connections to cluster mid-session) is given and something that cannot be changed.

In addition, the clients are unaware of any kind of clustering, so this aspects needs to be completely transparent. They also assume that any zone is available at any time and they can switch between zones freely and instantly without restrictions. The only physical separation that exists is between individual zones, but the transitions between cluster nodes within individual zone must be transparent.

So this is not about designing the latest greates MMO server. It's just a technical issue of retro-fitting an existing system.



You already made clear that the proxy server does the event marshaling and handles whatever routing is needed on the server side.


'Instantly' is something different when dealing with bubble worlds (teleport in and out, losing all dependancies with the old location, versus a transparent boundry system where the handoff needs to be as fast as possible and the player can still be effected by things on both sides of the boundry after the transition takes place.)


'latest greatest' is irrelevant. Im designing a system that ISNT a MMORPG and has significantly different specifications from a MMORPG (like really high powered AI that will eat most of the clusters processing capacity). BTW, some of the latest are still bubble-worlds because of the major problems (complexity, transition delays, complications, processing load) doing transparent boundries.


Im still not sure how your mechanism works with having multiple peer servers per zone. I would think that it would need centralized bookkeeping of some kind (for each zone) as well as for message dispatch within the zone's set of servers. If you have dynamic load leveling (within the zone) then even more overhead....

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this