The target system I'm investigating is :: a MMORPG with mutateable world ('infinte universe', auto-generation, on-the-fly level downloading from server, extendable world, on-the-fly bubble instances for auto-generated quests, deformable world, etc...)

This all is a requirement for something I want to see someday - player created assets for games (topic discussed several years ago in these forums) which would require more fluid client side world terrain data (no longer static levels, huge worlds etc..)

Ignore for now how/where that data is created. It will be based in the server and somehow get to the client in time to be digested and rendered for the player's 3D presentation.

---

Possible soution mechanisms:

Chunked terrain building blocks with 'dictionary/atlas' on the client already (alleviating on-the-fly large asset DL)
Bulk mesh data nd associated navmesh etc would be unitized and sent to client in patches of as background xfer.
Subsituteable attributes to vary these basic building terrain blocks (theme textures, axis rotations/fliops/scaling).
Easier than complete level data rebuilding. Neverwinter Nights had a system something like this.

Assemblies (precanned groupings) of prop objects that can be added/inserted as instances into terrain being displayed (discrete objects not part of the static terrain mesh) Discrete objects also assists in having more reactive world terrain.
LOTRO has something like this for player chars with so much varying clothing/horses that it takes many (many) seconds to transfer the dynamic data from server to client (very bad when large numbers of players in one place).

Player avatars/NPCs/vehicles currently are discrete objects but serverside these would be seperated more from the terrain associations (ex- no terrain triggers for NPC mobs). This allows for more complex (and variable) object AI (another one of my desired objectives).

Lots of 'templates' that have parameters to combinatoricly vary the look/feel/function of the 'building blocks' instances.
This allows specifics for instances to only need network send the base identity of the building block (already in the clients disk data store) and the Parameters -- a MUCH smaller set of data. Some programatic mutation is done to the final data before it is stuffed into the GPU.

Techniques to preload upcoming world chunks ahead of need (before the player will see them). Important -- the players can effect the base terrain beyond their initial state and thus all changes (deltas) will have to be patched over the base data (ie- I remove/detroy a prop or apply decals to indicate effects). The server keeps track of all these changes but they need to be transmitted to the client for CORRECT display of the world state.
You may be able to cache some of this world data client-side when the player walks away but other players may make additional changes (deltas) that will have to be corrected.


Auto-generation of terrain serverside (later sent to clinets). World terrain/assets is no longer static and can be expanded, modified as the plots procede, instanced, recycled .... Allows HUGE world with content created as needed
and more customization/variations (ie- not the same quests and locations that 1 million other players already went thru)


-------------------------------

Can any of the existing game engines do things like this???? (any talk of doing such features)

------

Actually the system could also be used for solo MP games with mutateable/generated terrain content ) -- maybe more likely than the MMORPG monstrocity .... A simpler mechanism that builds a static level but with many of the auto-generation possibilities.


Obviously 'detail' assets like textures/sounds/boneanimations/particle effect scripts/figure and prop meshes would also be downloaded, but again these can be done ahead of time (bundled as part of patches or in more frequent updates to add them to the client disk data store to be ready to be applied to the templated building blocks).

[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

Possible soution mechanisms:[/quote]

[/font]

[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

Mechanisms are cool, and all, but what problem are you trying to solve? I'd like to see the problem statement clearly first.

[/font]

[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

For example, one problem statement is:

[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

"A game engine that allows players to make persistent changes to terrain may require a lot of data download before a particular place in the world can be correctly displayed. I would like to minimize the time it takes from a player deciding to teleport to a spot, to the player being able to enter and walk around that area in the client in a shared instance."

[/font]


[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

Another problem statement is:

[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

"A game engine that allows players to make persistent changes to terrain will run into degenerate terrain edits, where players will attempt to tunnel infinitely deeply, build elaborate sculptures with evacuation masses and debris, and use terrain modification for griefing. I would like to minimize the impact of these problems without giving up player editable terrain."

[/font]


A third problem statement is:
"A game engine that allows players to make persistent changes to terrain may have players on high-end hardware make changes that are very detailed and render/simulate fine on their hardware, but a player connecting through a mobile phone will not have the same rendering or simulation fidelity; I'd like to support cooperative play without unnecessarily limiting the higher-powered hardware to the lowest common denominator; how can I do that?"


[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

The solutions are very different :-)

[/font]

I had suggested solutions (flavors of how client server components of a game engine would handle the things mentioned in the first line and was asking if any game engines out there (or proposed versions) might yet have similar general features.

[color=#282828][font=helvetica,arial,verdana,tahoma,sans-serif]

"A game engine that allows players to make persistent changes to terrain may require a lot of data download before a particular place in the world can be correctly displayed. I would like to minimize the time it takes from a player deciding to teleport to a spot, to the player being able to enter and walk around that area in the client in a shared instance."[/font]

[/quote]

This one is an aspect that would have to be handled (for teleport or just a login at that location). The term 'instance' I suppose would also cover a one copy world state bubble. Instance = 'realized' = expanded to working data...

The feature I did mentioned was having on-the-fly generation of these world locations (so a huge world can be stored in template IDs+coefficients and not as fully realized data) -- not so much having multiple instances of the same quest in the same world location so other players dont interfere with choreographed scenes -- which in some games is what they mean by 'instance')

Obvious solution is predownloaded (to client) of templated (bulk) data and smaller (near immediate)transmission of template mutation data(modifying factors/parameters/coefficients) with the full data being realization on the client for rendering/pre-collisiontest,etc. I mentioned additional cumulative delta patching of that 'base' realization being sent to change the final full data to reflect historic modification (player effectiong the world - causing deviation from the initial 'base' data ) to sync correctly with the state in the server.

The servers master data itself may be 'rolled' out of memory when vacated and collapsed back to the template+coefficients+deltas (maybe with additional ones accumulated via further player(s) actions) OR perhaps eventually deciding to just save it verbatum (full realized data image) once all those deltas get to be too many and there is little compression effected.

(I actually thought of game mechanics eventually 'healing' those player caused deltas so that it could again be reduced to the template+coefficient compressed storage -- especially is a HUGE world where its possible no player may ever visit it again and the full data storage is wasteful) Often there are also 'insignificant' changes that can be ignored and purposefully lost by the re-compression (only significant deltas would be saved)

======================================================================================================

[color=#282828][font=helvetica,arial,verdana,tahoma,sans-serif]

"A game engine that allows players to make persistent changes to terrain will run into degenerate terrain edits, where players will attempt to tunnel infinitely deeply, build elaborate sculptures with evacuation masses and debris, and use terrain modification for griefing. I would like to minimize the impact of these problems without giving up player editable terrain."[/font]

[/quote]


"degenerating terrain edits"?? hmm distortions of the mechanics that cause problemaitc situations. Minecraftisms. Well the system I would design wouldnt allow such things. The server arbitrates all compound modifications and cases of hitting limits handled logically (crossing zone boundries and such covered). So is solved with proper game mechanics and not really part of the data replication and update transmission mechanism.

World state Deltas conversions/compressions are run on locked down/settled data when 'rollout'/collapsing takes place (no players present, no unresolved modifcations pending).

Server on-the-fly sync 'save' images though -- use briefly locked down zone for snapshot or somesuch (compression from second buffer).

======================================================================================================

"A game engine that allows players to make persistent changes to terrain may have players on high-end hardware make changes that are very detailed and render/simulate fine on their hardware, but a player connecting through a mobile phone will not have the same rendering or simulation fidelity; I'd like to support cooperative play without unnecessarily limiting the higher-powered hardware to the lowest common denominator; how can I do that?"
[/quote]

Mismatched hardware target for client machine problem. Solution dont allow(or disuade attempts to) play on systems below minimum requirments.
Simulation on server goes thru as normal (full detail), sub-par hardware client may choke on mass of data thruput (heh, may not fit the dictionary/atlas disk space needed for the templates or load them fast enough) or have enough CPU for client realization/expansion processing or even proper/timely rendering --- as they fail already on many MMORPG games.

Simplification of data features already on some existing engines.but if 'very detailed' game data being presented are important to game (faces in LA Noire??) are LOD'd/simplified out then again sup-par hardware has to be identified by developers and user told up front that it wont be workable.


Part of a MMORPG (of this kind Im asking about) actually could have RTS elements which would have 'views' into the game with tablets/palmtops for certain (non-3D FPS) aspects and would provide a suitably simpler RTS 2D interface for monitoring/controlling NPCs tasks/projects and minigame puzzles and ingame mail/AH -- while the player is not on their highpowered client box.

That brings up a whole other issue of (advanced AI driven) NPCs running sometimes in full 'realized' detail (when players are near) and other times in an 'abstract' simplified manner (still carrying out tasks/projects but not needing the full world detail where they are - server zone's simulation may be running in generalized/abstract mode to lower processing load)

===============================================================================================================

Instead of doing a full blown MMO, I'd look at maybe only taking control and hosting player data and possibly lobbies, while letting users host their own worlds so you can avoid legal issues with user generated content. You could even let players have portals in between their worlds so the game seems massive.


I'm working on a fully dynamic game like this at the moment but with no multiplayer yet. If I do add multiplayer I will probably do what I said above. If you don't want to get sued I'd suggest doing the same.


As for the problem of getting the deformed content to the user... what I'd do is download all dynamic data on the first time playing then just keep changes in a buffer for each client then stream the ones the player is closest to first and then the rest.

You'd also have to keep an offline change buffer for each player. And again, just stream the closest ones when the player logs in and all the rest whenever you need to.

User generated content is unpopular since it's a target for lawsuits. When someone makes their character look like spiderman, the IP owners will come knocking demanding licensing fees. Similar with music. Then there's users posting undesirable content. Not a problem in itself, but may require oversight due to laws and age restrictions may be tricky to enforce.

As with most things today, technical issues take back seat to most everything else, it also explains why the initial 2000-era push in that direction completely died off, especially considering SecondLife.

And as with everything user generated, problem is building a community, which is where the interesting problems lie. "Huge" is also an interesting gimmick, but so far it has yet to be used efficiently, especially for curated content. For anything social, size must be just right, just like humans are built to interact with ~150 people at a time. These sizes appear to be mostly within the limits of hardware today.

Hardware itself is also less of an issue, just about everyone has settled on one or two baseline profiles (perhaps console vs. PC). With increased fragmentation and mix of mobile vs. laptop vs. desktop, offering one-size-fits-all ins't viable anymore.

The legal issue was one I actually put at the top of the 'difficulties' list I posted a few days ago.

I considered it years ago and figured the 'company' would have to have a good vetting process, and to keep the labor costs down
would have the players do alot of the up-front grunt work pre-vetting assets first and have well spelled out asset requirements/specification. With some kind of rewards for finding bad content that were submitted (need so way then to keep people from intentionally submitting bad stuff so it could be 'caught') and a 'blackball' system that would slow down submissions from problematic posters of assets (you wouldnt be allowed to submit more complicated things til after you had a track record with the easier stuff - since things like scripting also take more skill and you dont want to be flooded with inept content). The other aspect was the collaboration model that could pool skills and maximize reuse/variations.

The vetting would have online player reviews (with sufficent exposure time) and after it passed that (also includes basic validation tools - also player run) then it would go to inspection by company paid inspectors before full sandbox testing (real server exposure if it couldnt be done at an earlier stage to find hard to spot coding bugs in scripts) Basic assets - textures/sounds/music/text which might have copyright infringements would usually be spotted much easier. One advantage is that bad assets that slip by all the vetting can immediately be pulled/blocked by the same update mechanism.

Im not sure but actual ownership of unique 'art' being validly submitted becoming 'company' use legal (within the game context) ---- can a EULA (or equiv) boilerplate over that endcase sufficiently???

---------------

I havent looked at Second Life in years (last saw it just as they were banning gambling which I figured would elimiante about half their userbase's interest in playing. They didnt have any good set of 'templates' then to get creators started, and no purposeful collaboration model (since selling things in game was a key aspect).

-------------

'Everything user generated' --- the particular candidates I was thinking of would either be new MMORPG of existingg solo games to have some 'seed assets' to provide playable areas from the start (have a leg up on limiting some asset expenses) . Else the company would have to generate NEW 'seed' assets to geth things going and then expand with the player stuff, That actually might not be awful because part of the process would be developing those 'templates' Ive refered to before (while creating their set of initial assets) that greatly assist in players creation (alot of stuff is variants with textures or tweaking shapes existing bone meshes and changing coefficients for variation of weapons and such using the existinggame mechanics. Etc..

Using previous (solo) game info may be problematic if either different contract work for the assets didnt specify reuse/full rights when handed over -- or similar legal issues. You would also hope the data conversion would be straight forward (MMORPGs dont always use cutting edge engines and reusing a new close/same one for the MMORPG may be possible. Solo games do rely heavily on static 'levels' and chopping up those static level meshes to create individual object meshes may offer difficulties (or may be quite easy for at least that part of it) Behavioral and other Object oriented data will have to be created anyway (again templates to maximize reuse on common aspects).

-------------

Community - I have played in a recent MMORPG and was very disappointed with the poor and decreasing community aspect as compared to long ago with games like UO. The game Im using as an example would be built around lots of player cooperation -- hopefully freeflowing enough (bringing back visible talk text maybe ?) and with mechanisms to facilitate 'community' (in-game newspapers/bulletin boards, short term quest contracting etc..)
The players would have 'lackey' NPCs who can go perform tasks and so they could interact even when the player himself wasnt there.

The old NxN problem with too many players(clients) all in the same place - I dont know if you can yet architect around that to much bigger N numbers.

Visible faux-community would be an attempt also to have alot more NPCs that are reactive and move about more doing things (like GTA San Andreas )
so central areas wont look like the typical MMORPG ghosttowns.

-----------

Console is an issue that could be a sticking point since the companies pretty much need to offer it these days. Do THEY have the capacities (I mentioned large disk dictionary/atlas space requirements may be needed to get this 'on-the-fly' generation to work on the client end, ditto for network download stream WHILE the game is running (maybe?)) Mobiles might be cut out. Though some of the RTS component of such a game (one Im looking at as a candidate) could work on the tablet/palmtops with 2D (may not on other genres).




--------------------------------------------------------------------------------------------------

If anyone is interested Ive been posting at a company game site with ideas for a specific game to make into a MMORPG -- believe it or not, its for Bioshock. The new one (BS Infinite) they are doing (out this year sometime) isnt the same plot context and actually borders on Harry Potter style fantasy.

http://irrationalgames.com/community/forums/bioshock-general-discussion/bioshock-first-person-mmo



I was composing more blatherings for it on 'the building block concept' (templates with interchangeable subcomponents - themselves based on templates) and on how the 'auto-generation mechanism' might work for the expandable world idea. I might post those tommorrow.

Instead of doing a full blown MMO, I'd look at maybe only taking control and hosting player data and possibly lobbies, while letting users host their own worlds so you can avoid legal issues with user generated content. You could even let players have portals in between their worlds so the game seems massive. I'm working on a fully dynamic game like this at the moment but with no multiplayer yet. If I do add multiplayer I will probably do what I said above. If you don't want to get sued I'd suggest doing the same. As for the problem of getting the deformed content to the user... what I'd do is download all dynamic data on the first time playing then just keep changes in a buffer for each client then stream the ones the player is closest to first and then the rest. You'd also have to keep an offline change buffer for each player. And again, just stream the closest ones when the player logs in and all the rest whenever you need to.

Im not doing it myself, Im more trying to convince game companies to improve MMORPs in the ways Ive mentioned (the current crop havent significantly changed from 10 year old methods and in many cases move backwards)


There is a game called Neverwinter Nights that had(has) a similar model to yours with a actual server (I played around with it many years ago. They supported upto 64 players per server (effective was more like 30) and they did have players do portals between multiple serever to extend their capacities and world size.

They actually used a 'chunk' system with precanned terrain modules (few parameters on the modules, but they did allow placeables and the usual triggers and spawns and NPCs with dialog trees/questgiving). People published their 'world' sets but most were truely lame and only a few seemed to put the thousands of hours of development needed to create good interesting/cohesive ones and THEN they also had to moderate their servers.

They also did have player created/modifies assets that added alot to the game (more terrain modules, lots of clothing, placeables, NPC/monsters figures and such - animations?) and distributeable serverside scripting (ex - a substitute scripted battle AI for monster opponents alot better than the company supplied one was ). They did at that time supply at least twice large asset set patches/replacements (which you had to have to run world created using those extended assets sets).


They had some halfway good tools to build worlds (out of the terrain modules plus the usual trigger/spawn placemenst and more) and an interesting NPC dialog tree editor (imbedded animations for the NPC and player attribute adjustments and other activations)


A significant problem was that the game NEVER (as far as I know since I havent looked at it for years) had a Program Interface (DLL or otherwise) to link in native code. It all ran their lame/limited slow scripting language. Players did cleverly reverse engineer some of it and actually added a persistant bank system (saved containers to SQL) triggered by some trap that intercepted in-memory buffers). A program interface would have made server-side processing improvemenst by players much easier and could have used the speed of additional server boxes to run NPC/Monster AI and smart spawning and to cross between server worlds while in-game. It was actually a mystery to me why then hadnt thought of it and never added it when many players were calling for it over the space of several years. Even simple data transfer via TCP routines via routines their script langage library (to allowtie into native C++ processing on a second/additional boxes) which would have been dead simple to do, they never did. (idiocy in my book).




You may want to look at creating a server component to act as a ' world change' clearing house so that individual clients being lost doesnt take the change data with them. Failover server transfer is a typical feature so that if 'the server' machine is lost all the game isnt lost, though recovery still might take half a minute for the primary serverr replacement to sync with all the clients and signal the transfer of 'serverhood'

I am actually implementing a voxel based terrain in our MMO (Abydos Online).

I wrote a simple C module, 1000 lines of code (500 for functionality and 500 unit/integration tests) that reads/write voxels to disk.

Here is how it works:
The terrain is called a volume.
A volume is divided into regions (default 8x8x8 regions).
One region is divided into blocks (default 16x616x16 blocks).
One block is 32x32x32 voxels (1 voxel = 1 cubic meter).
Each voxel is 4 bytes, 1 byte density, 2 byte material and 1 byte reserved.

When terrain is initiated, the C module creates a directory with one file per region. If the region is homogenous, only that file exists which contains the voxel for the whole region (for example all air, dirt, water etc). The filename would be "region_x_x_x.dat" where the x is replaced with the x, y and z position.

If a region is not homogenous, a directory "region_x_x_x" is created which contains the blocks, by default 16x16x16 blocks result in 4096 files, each of this file is named "block_x_y_z.dat". If this file will either be 1 voxel (4 bytes) in size of a homogenous block or 32x32x32 voxels (128 kb) for a mixed block.

A block with all the 128 kb of data is referred to as a chunk.

Currently working:
When a player logs in, the whole terrain is read and sent back to the player, regions and blocks are sent one as one TCP packet back to the client, chunks divided into parts of 8kb each and reassembled in client.
The client stores the data in the same structure as the server but also parses all regions/blocks/chunks and build static terrain geometry of this.

Todo:

• Terrain versioning, each region/block/chunk should have a version number (2bytes, 4 bytes, 8 bytes?) that is increased per change. At one point the max version will likely be reached and you will have edge cases where players that should download updated terrain does not. A straight forward solutions is to have a dirty flag forcing all players to re-download the whole terrain and restart versioning. (Other solutions probably exists, but I think this is the best one since it does does not have any edge cases)
  
• Each region/block should have a sort of level of detail (LOD).
  
• Server should have two caches, one for reading voxels and one for writing.
  
• Reading buffer - When a read request, check buffer first, if not in buffer read requests terrain and its nearby terrain data and store in buffer (memory).
  
• Writing buffer - When writes requests are done, don't actually write at once but wait for some time possible save up some writes and do them all at once.
  
• Server should propagate changes in the terrain to all clients (at the moment, clients have to restart to get updated terrain).
  
• Server/Client should compress chunks before storing on disk (bzip2/gzip).
  
• Writing requests to the C module should be replicated to other areas that "run" the same area, making it scale horizontally. This allows servers to come out of "sync" with their terrain. A sane voting protocol (probably using the versioning if possible) would have to be implemented to allow servers that has gone offline to sync up with the other servers.
  
• C module improvement could be done by some threading (I will try to avoid this as long as possible).
  

Not needed:
Clients does not need to request writes to the server side terrain since all logic run on the server. (For less "safer" solutions, this is just a matter of adding a new message type on the protocol). Write requests that are issued from the clients are possible more open for exploit.

Problems:
Server disk space not much of an issue, our plan is to have 10x10 areas of 8x8x8x -> 16x16x16 -> 32x32x32 voxels (256 GB of data per area in worst case). Which means 25 terra byte for the whole world.
Client side, this would of course be a problem, but worst case is highly unlikely to be achieved, most areas will likely have many homogenous parts, for example half the area would be air, other half dirt/rock and most of the mixed chunks would be the surface or mines. Some gameplay mechanics could be implemented that tries to make parts of the terrain homogenous under certain circumstances.
One could have a special storage cache rules for the client, for example not allowing storing more than X GB of data on disk, freeing up space as new data is written (size could be set by the players themselves but should have a minimun value for sanity). This also requires clients to request syncing a full area since the logic on the server would have this client information.

I am actually implementing a voxel based terrain in our MMO (Abydos Online).
...

If possible you may want to look at optimizing your compression to make the most o fthe likely mostly empy Z (up down) direction
(probably by ordering data Z major so largest compression blocks are likely empty and thus compressed to virtually nothing leaving the more
detailed (lower) surface XY plane layers to be clustered together). Disk<->memory (and network) interactions because of their slowness make even minor speed improvements via smaller data image size worthwile.

Another issue (which may or may not be relevant) is server save image generation with such large data sets (mine was frequent ~4GB which still had significant copy time) Running in a journaling mode while the base image (file set) was locked down for verbatum copy and then doing catchup after the lock ended added many complications to the server data mechanism,

Seperate chunk files within the world image might be simpler, but you may find a large monolithic file with Raw Disk IO will speed it up more than a little.

--

I also have a 'chunking' scheme for my 'big' project (its more conventional flat map terrain mesh+ local objects flavoring) with the clients having a 'window' (15x15 chunks of progressive LOD) view on a large world (1024x1024 chunks). It included seed coefficient based terrain+local object auto-generation (and a delta change list and preferred collapse back to template+coefficient). Unfortunte the simulation has requirement of absolute cohesion/completeness of server data saves which made the system much more strict and complex to achieve that goal.

Simplified question then:

Do any of the current game engines have features for 'on-the-fly' terrain chunks being sent to clients to accomodate 'player changed worlds' ??
(not just the usual objects moved)