Distributed server architecture for load balancing

Started by
50 comments, last by _winterdyne_ 18 years, 6 months ago
A-P, as you suggested, each event has a radius of effect, as do sensors for event types.
The aggregated sensory range of the contents of each POR (a variable size AABB) is used for coarse collision testing between effect area and possible sensors. This is done by a parent POR. Knowing maximum radii for various event types allows culling based on the PORs' actual (physical) sizes so there's an early-out method in play.

You are also correct in your observation that high adjacency can cause more overhead regarding neighbourhood. This is partly why a regular grid based spatial model is not being used (although it could be). If the majority of those grid squares don't contain anything 'interesting' to cause the player base to spread out, the majority of the player base will centre around the provided content. Also consider, this may be used for a true 3D game, such as a MM space sim - using a regular grid based system in such a game has its disadvantages - not only because of the extra dimension of free movement, but also, as mentioned because of the big empty spaces between any meaningful content. This is why I refer to the system as a *relevance* graph, rather than simply as a spatial divisioning system. I want to be aware that it is the organisation of *content* and its delivery that I'm dealing with, not merely dividing up a landscape into regular sectors.

In terms of crossing server boundaries, message-aware objects are intended to be placed into a transitional state. The object will be temporarily locked, and messages for it will be cached by the master server, and inserted to the start of the new queues once the object is reactivated. The object will experience a short additional lag as this occurs.

Anyway, this'll be my last post on the subject for a couple of weeks, since I'm off on holiday to the Algarve today! Thanks to all for some thought-provoking and helpful answers and I'll see y'all when I get back.

<virtual postcard>

Winterdyne Solutions Ltd is recruiting - this thread for details!
Advertisement



Actually I was considering the grid as 1) just an example of the possibility of large numbers of N-way 2D/3D adjacencies (versus some games that are simply a network of portal connected bubbles) and 2) as a universal coordinate system that distances could be calculated with (to impliment the culling).

Another problem that you may not need to solve is: where you have vehicles (like a ship on water) that can move and cross area boundries (and even exist in 2 or more simultaneously) and which has its own local relational reference system seperate form the world's -- events have to cross boundries that change , translate AND rotate....
For mobile hierarchies such as ships etc, there is a roaming subclass of POR.
Within the mobile hierarchy operation is exactly as per normal. There are also subclasses for pure hierarcical and physical PORs. In my example in this thread, the _root and game world PORs are pure hierarchical PORs, and the rest physical since they may contain stuff. Mobile objects (including mobile PORs) can only exist within physical PORs, but a mobile object is in effect a pure hierarchical POR. Thus, a mobile cannot contain a mobile.

The mobile node itself is treated similarly to mobile POR 'limpet' or actor, in that it does NOT expand the physical extents of its parents (which fixed hierarchy POR's do). Its parent link is changed on complete exit from its current parent and is reselected primarily on volume inclusion (most volume included in a hierarchical fixed POR). I have a planned variant which will take velocity into account, and select a parent based on occupancy over a certain period of time.

Note that mobile POR movement, similarly to mobile actors (as opposed to placement / teleportation) is limited to neighbour relations.

Rotation of a POR (can be applied to any POR, but in runtime only to mobiles) dirties its contents positions, which triggers an AABB update of the parent. This 'dirtying' does NOT update the AABB of the parent of a mobile POR, but will update the mobile POR itself, which may result in it being transferred to a new parent fixed POR.

Message handling from a mobile POR is via its fixed parent. Since the neighbourhood relations of the mobile (outside of its internal hierarchical links) may cause message repetition or duplication, the fixed hierarchy is used since the neighbourhood relations there are not generally variable.

Because of this a mobile can only exist in ONE fixed POR at a time, but may be affected by nearby neighbours of that POR.

Does this sound about right to you folks?

Winterdyne Solutions Ltd is recruiting - this thread for details!
Why wouldn't you just keep your entire world in one big coordinate system, and use that for all simulated entities? Why use this ship as a point of reference, when you can just express both the ship, and the things on it, in the world coordinate frame?
enum Bool { True, False, FileNotFound };
The coordinate system is global.

The reason for the hierarchical structure is message culling at a node level.

In this way, a mobile entity may actually contain its own effective hierarchy - say that the ship contains corridors and rooms within it. Neighbourhood relations can constrain message traffic within that hierarchy.
Winterdyne Solutions Ltd is recruiting - this thread for details!
Quote:Original post by hplus0603
Why wouldn't you just keep your entire world in one big coordinate system, and use that for all simulated entities? Why use this ship as a point of reference, when you can just express both the ship, and the things on it, in the world coordinate frame?



The ship moves (translation and rotation) and you would have to constantly update
the coordinates of all the objects on the 'mobile' as well as all the terrain geometry (like the decks, passageways etc...) for the objects on board to interact (they generally will interact more with other things onboard than with things off the ship). Events that cross the boundry would have to be converted from the local coordinate system to that of the parent (which the ship sits on) or PARENTS
(when the ship crosses one or more boundries at that level of the hierarchy).

There isnt any simple way to do this and you will have to eat the extra calculations.


Yet another fun case is having objects on 2 ships interact with each other.
The topic author is working on formalizing the way the events are passed between
different referents (including where the other 'referent entity' exists on another machine).

Quote:
(snip)
Events that cross the boundry would have to be converted from the local coordinate system to that of the parent (which the ship sits on) or PARENTS
(when the ship crosses one or more boundries at that level of the hierarchy).


The global coordinate system fixes scale and orientation, not origin - this is always 'local to parent' - ie. the parents 'real' position is 0,0 for the children.

The AP is correct, assuming that a mobile built using a mobile POR is designed for constraining messages inside itself - otherwise Hplus is correct and less work is involved in having multiple mobile 'limpets' (a term I use for occupants of a POR) being moved together - most likely through the use of a 'congregation' mobile. Mobile POR's are not designed for things like rafts, where the contents are visible, but physically tied. They are designed for discrete hierarchies which roam within a larger hierarchy - like cruise ships.

Quote:
Yet another fun case is having objects on 2 ships interact with each other.
The topic author is working on formalizing the way the events are passed between
different referents (including where the other 'referent entity' exists on another machine).


This is actually trivial and is best examined by a 'bad case' scenario, where a fairly large amount of traffic may be generated across a machine division. I'll avoid going into too much detail about synching the various steps of the process and the worker threads that deal with inter-server communication, since its all fairly simple.

Assume we have three physical POR's, Land, Bay and Ocean, forming part of the divisive POR, Coastline, which is part of GameWorld.

Land is hosted on ServerA.
Bay is hosted on ServerA.
Ocean is hosted on ServerB.
ServerA is the Master Server for this cluster, and also hosts Coastline.

Land neighbours Bay, Bay neighbours Ocean.

Bay contains mobile POR ShipA, partially in Ocean.
Ocean contains mobile POR ShipB, partially in Bay.
ShipA has two occupants, PCA and PCB.
ShipB has a single occupant, PCC.

For the sake of argument, Land will be out of range of 'shout' events from both ships.

As mentioned, a relevance update occurs in 3 stages, the logical update, the handling update and finally the client update.

Step 1 - Logical update:
ServerA syncs ServerB.

PCA (on ShipA) shouts "Ahoy there!", for the sake of argument.
This generates a 'PCA shouts "Ahoy there!"' message.
The PC, being a limpet, deposits its message directly to its owner, ShipA.

ShipA being mobile, forwards the message to its parent, Bay. The current location of ShipA and the origin of the message give the location of the event relative to Bay. Mobile PORs cannot be responsible for message distribution, since their extents may cover varying portions of the fixed hierarchy.

Bay is now responsible for dealing with forwarding the message to all PORs that are in range of the event. Our shout message expands outside the area occupied by Bay, and so must be passed to the parent, Coastline for handling.

Coastline recieves the message and calculates the event origin relative to itself. In this case the message and its radius of effect lie entirely within the division governed by Coastline. The message is queued for handling. GameWorld need not be informed of the event.

ServerA being Master, checks that no slaves have pending events this cycle before continuing.

Step 2 - Message handling:
ServerA syncs ServerB.

On ServerA:

Coastline checks its child nodes' extents, Land, Bay and Ocean. Bay and Ocean are covered by the event, Land is not. Note that Coastline knows nothing about the contents of Ocean, since that is on a different machine, although contents can be queried for.

Since Bay is local, a handler function is called directly with a pointer to the message. Since Ocean is not local, a network message is created with a copy of the message data, is encrypted (if internal encryption is used on the cluster) and added to the server send queue. A worker thread despatches these immediately. This is efficient on dual processor systems, but has no advantage over waiting till the end of this step on single processor systems.

Bay handles the message by passing it to all its contents' message handlers (since it is a shout). ShipA receives the message and behaves similarly, passing it to PCA and PCB.

Both PCA and PCB handle the message by adding it to their network send queues.

On ServerB, simultaneously to the above:

ServerB gathers update messages from machineA using a worker thread, and processes them immediately, in addition to handling any events generated and handlable internally (in this case none this cycle).

The shout message is received and passed to the default slave root node, which contains Ocean. Ocean passes it to ShipB, which passes it to PCC, which places it in its send queue.

On ServerA:
ServerA informs its slaves there are no further events from it this cycle. This puts the worker thread on the slaves to sleep.

Step 3 - Client update:
ServerA syncs ServerB.

On ServerA and ServerB, simultaneously:
Send queues are processed cascade fashion. PCA and PCB receive messages from ServerA, PCC receives messages from ServerB.

The worse-case scenario where the shout is sent from ShipB works similarly, but the shout message actually crosse the server boundary TWICE, rather than once, since Coastline is remote as far as Ocean is concerned - the message is passed up across a network connection, rather than just down as in the example above.

Does this make sense? Maybe I should do a diagram?

[Edited by - _winterdyne_ on October 11, 2005 8:43:47 AM]
Winterdyne Solutions Ltd is recruiting - this thread for details!
Quote:The ship moves (translation and rotation) and you would have to constantly update
the coordinates of all the objects on the 'mobile' as well as all the terrain geometry (like the decks, passageways etc...) for the objects on board to interact


When the ship lurches, you want the player to lurch, too, not just stay rock solid on the ship's deck. In fact, gravity doesn't change when the ship moves, but the ship changes. If you run an actual physical simulation, it would be simpler to simply change the ship, keeping everything in world coordinates, because the physical simulation will take care of everything. ("having to update all the objects" is not a problem -- because they are simulated, they are updated every frame anyway)

If you run a non-physical, old-game kind of simulation, perhaps using an old-school scene graph system, then isolated coordinate system islets would make sense. But hopefully nobody's actually building yet another one of those in this day and age!
enum Bool { True, False, FileNotFound };
Quote:Original post by hplus0603
Quote:The ship moves (translation and rotation) and you would have to constantly update
the coordinates of all the objects on the 'mobile' as well as all the terrain geometry (like the decks, passageways etc...) for the objects on board to interact


When the ship lurches, you want the player to lurch, too, not just stay rock solid on the ship's deck. In fact, gravity doesn't change when the ship moves, but the ship changes. If you run an actual physical simulation, it would be simpler to simply change the ship, keeping everything in world coordinates, because the physical simulation will take care of everything. ("having to update all the objects" is not a problem -- because they are simulated, they are updated every frame anyway)

If you run a non-physical, old-game kind of simulation, perhaps using an old-school scene graph system, then isolated coordinate system islets would make sense. But hopefully nobody's actually building yet another one of those in this day and age!





I suppose if you are going to do physics like that you could try a universal coord system that updates constantly. Unfortunately you will have to decide where you want the 'realism' to stop (in order to make a game that doesnt require a supercomputer to run in real time). Sure you could have the player lurch about (and check all the friction effects that keep objects in place most of the time) but what of the entire structure of the ship/boat. Do you want to have to calculate all those structures effects by every force upon them to calculate all the transformations for positions (culling methods would help this some). Its probably more cost effective to apply the various 'lurch' forces to the player and other 'moveables' within a local coordinate system to minimize the CPU load.


We are still stuck with those 'old school' mechanisms because the game worlds have grown larger and the detail level higher (more objects) enough to require running the simulations on multiple machines (as per the topic).
Quote:Original post by _winterdyne_


Does this make sense? Maybe I should do a diagram?





Yes, I think I followed most of it, but diagrams would make help visualization.




Also are there any additional complexities for an event that dont just have a simple origin and spherical effect (shout) but have more directional interactions (ie- an arrow fired) that may need to do a collision check/ LOS (line of site) and/or non-instantaneous (a traveling arrow) that itself moves over time and ack! may have secondary effects (like being visible/viewed by other obects as it moves...).

This topic is closed to new replies.

Advertisement