It gets nasty. Each chunk needs to maintain a border area (to a depth of any significant visible actions/events) for all of its adjacent chunks. You then maintain a shadow copy there of any significant objects within that border area of that other chunk (you have some kind of subscription to channel updates to THIS server machine...) . Every time something happens to those potentailly visible objects in the other chunks, the current chunk has those shadow objects updated with whatever significant state/event happenings are specified by your game mechanics.. You can now do basic visibility checks and projectile pathing and send client update info )object positions/events) without having to constantly transmit requests for that info to the other chunk's server. Needless to say, that the quicker the updates are maintained between servers, the more accurate the interactions will be.

When a players client institutes actions against a shadow object that action event would be (pre?) routed to the other server to take effect (it may would maintain full details to do all game mechanics calculations including with object beyond that border area which may have effects) Results of the inetactions then get forwarded to the subscribing chunks.

Chunks completely out of range of a players viewing could be LOD'd down in running priority (with catch up and switching back to full detail when the player does move within range).

Now it gets fun when you have chunks with players in them being 'high priority realism' and others (adjacent) with 'low priority abstraction' and beyond them chunks being rolled out or being ready to be rolled into main memory when a player gets close enough. There then would be breaking/establishing of the 'subscriptions' in a fluid manner.

Note- at a grid 'corner' you can have you object in three other chunk's "shadow" lists (or worse if you have irregular chunk layouts)

A little more fun is : if you have seperate CPU cores running the chunks and having the update pipes either go to a whole nuther machine or just (shunt) to a different core/process space on the same machine. At one time I did some research on lockless queues to try to minimize overhead for this kind of thing. But then I also had (many) seperate server AI nodes to run NPC units, and like a player's client, each 'smart' object could traverse between chunks and potentially maintain its own world view of many chunks it had visible.

Having built and maintained a system like this, I'd like to suggest that trying to do it on top of an existing physics engine will be a challenge.

It can be done, but if I were to build it "for real" again, I'd either re-build my own physics engine, or use some open source and significantly re-architect it to suit the problem.

I would possibly also not use the border/cell method, but instead use a system where multiple servers can share entities in the same instance, and simply make a hard rule that entity/entity interactions must have at least one tick's worth of latency, so that effects from entities can be transmitted to all other servers that serve entities in the same area in time.

If you are a small business/studio or even a hobbyist, I would suggest building your world as separate areas (city districts, islands, whatever) and use some kind of "neutral, no-effect zone" between the areas, so you didn't have to worry about A shooting at B. Yes, this changes the game design a bit, and does so in the interest of actually finishing a game. If it turns out to be very successful, perhaps you can add the hard technical challenge in a later update/version.

"but instead use a system where multiple servers can share entities in the same instance,"

Not sure what you mean by this.

You could get rid of the borders on the system I describe (if its too complicated) and just get 'shadows' from the entirety of each of the adjacent 'chunks' (the chunks being spread out on multiple servers and THEY do the bookkeeping for the world)

To do behavior AI decisions for NPC objects,

execution of client avatar commands/actions,

and for all the object Bookkeeping (arbitration of effects/command action results)

All of those would need authoritative processing locations (however distributed for entities involved)

and being distributed for scaling, will have to keep others (entities) informed with COPIES of each authoritative decision/outcome(and current states)

it has to be a prearranged 'push' flow because you dont know what has happened elsewhere to know to 'pull' anything

unless you have fully deterministic game mechanics you cant arbitrate equivalently in several locations for every decision/outcome

"but instead use a system where multiple servers can share entities in the same instance,"

Not sure what you mean by this.

What I mean by that is that I'd probably allow multiple servers to serve the same area, and be "masters" for different entities within that same area. A cheap ghost of each remote entity needs to be created on each server for each remote entity, but this can be very, VERY cheap -- it doesn't even need physics, other than whatever the collision shell is.

Then, when there is player/player interaction, you simply make the rule that interactions take one tick -- this gives the interactor time to send the message to the interactee about the interaction for the next simulation tick.

This way, I believe I could get to denser areas than an area-derived server split, because many servers can overlay various areas, and if you need more entities, you add more servers, and don't have to worry about the area.

The problem with area-based servers that use physics-based ghosts is that there exists an upper limit to the density (in players-per-square-meter) you can achieve, because you'll end up with too much load just serving border areas if you make each area-based cell too small.

OK, like I saw likely solutions : a 'chunked' Bookkeeping server(s) (at least to maintain data if not decide/arbitrate changes to the data), and other 'servers' to control NPCs/props and then the front ends for clients.

The physics 'events' (living whereever - now on their own server subset....) would be their own 'temporary entity' which are invoked and pull data from all the other involved entities (terrain data, action originator, targets directly/indirectly involved whose states might be in flux from simultaneous effects in the simulation...). I would expect you would need a one 'tick' delay anyway to let all the different factors being involved to be consolidated so a proper decision of the result would be made and then the result posted (possibly preciptating follow up events from sideeffects - cascade of secondary events).

That interaction 'tick' itself could be divorced from alot of the other timings happening on servers(and of course clients), but fast enough for a low feedback delay rendered on the client end (results of their actions visible without too much obviously/intrusively delayed time)

Depending on game mechanics -- like long range projectiles crossing many 'chunks', alot of data has to suddenly be pulled between servers (OR (when appropriate) a 'contination event' like the next segment of a ballistic path that is small data but then will be executed on that chunks locally (like collision check against the chunks terrain mesh and set of dynamic objects).

Either alot of remote data has to be pulled, or maintained cached (with update flow) or the interaction has to be farmed out to the locality(s) it immediately interacts with.

SO its a balancing of how shallow/simple you can make the 'ghost/shadow' data - very frequently used data like position/appearance/current animations might be 'echoed' in the shadow/ghost on a server - just to trigger interactions (and like a client front end that might serve hundreds of clients all looking at a similar scene -- so each doesnt have to indpendantly pull across the same cross-chunk data - instead it is cached locally (once) and shared between them all) Still a regularized subscription mechanism in 'chunks'...

More detailed data which is used only for infrequent interactions (like physics) would then be pulled only when needed.

-

Ive been examining mechanisms like this because of my interest in heavy weight AI for NPCs who would require their own server nodes to run on and thus would pull alot of data (much like player clients do) being highly reactive to situation changes when near the players, and have to maintain a continuous local world view for the AI to work upon. Server-to-server network traffic from that is greatly increased (smart objects of the type Im thinking of need to continuously see ALOT of their changing surroundings) thus my interest in optimizing server-to-server network data flows (an obvious optimization is players 'sidekick' AI NPCs actually running on the players client where they have direct access to the same data the player is seeing, besides farming the AI processing out to far more processing capacity than centralized servers)

the problem you describe comes from using a level based design paradigm. you may want to research how large simulators are made. games like red baron 3d - where the game world is the entire western front and they model every aircraft in flight along the entire front from switzerland to the channel. or silent hunter where the game world is the entire world, and they model every ship in the pacific theater of operations (maybe more, never left the pacific). or my current project CAVEMAN, where the game world is 2500x2500 miles in size, with NO load screens whatsoever. and yes, you can shoot across terrain chunks and 5x5 mile map squares with no problem.

i really should do an article about the evils of level based design. doom is not the only way to build a game, and is a very sub-optimal way to build a simulation (as you're discovering).

Two comments:

1) There are very legit game design reasons to use levels for certain designs. Having players be able to escape a too-hard monster by simply changing levels to run away is a fine design choice, for example. if the world is fully roaming, there's really no clear way to "run away" when you get in over your head.

2) Regarding your signature: "Changing textures" isn't actually all that bad. "Changing formats" is worse. And "changing vertex formats" is even worse than changing texture formats. Changing vertex formats may very well require a re-compile of your shaders, which is more expensive than pretty much anything else. Changing texture formats likely requires a stall of some sort, so fairly expensive. Changing from one vertex buffer to another with the same vertex format and stride, or changing from one texture to another with the same size, pixel format, and MIP layout, is likely to be very cheap these days.

1) There are very legit game design reasons to use levels for certain designs. Having players be able to escape a too-hard monster by simply changing levels to run away is a fine design choice, for example. if the world is fully roaming, there's really no clear way to "run away" when you get in over your head.

therin lies the difference between a game and a simulation. in a shooter, game play comes before realism. so the fact that your opponent is no longer modeled, because the game only simulates the active level, and you just changed the active level map, leaving your opponent on the previous level, is ok. not very realistic, but ok (matter of opinion there actually, i personally would consider it unacceptably unrealistic - but then again, i'm an old school hard core gamer.). In a simulation, if you get in over your head, you're _supposed_ to die.

re the signature:

the quote refers to runtime overhead costs only, and is drawn from experience with dx9 fixed function. however, it seems that the bind times in later versions of directx have the same relative order of overhead cost, IE binding a new texture is slowest, etc.