View more

View more

View more

### Image of the Day Submit

IOTD | Top Screenshots

### The latest, straight to your Inbox.

Subscribe to GameDev.net Direct to receive the latest updates and exclusive content.

# Continuous Asset Data Streaming - issues/problems with

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

17 replies to this topic

### #1wodinoneeye  Members

Posted 14 January 2014 - 11:36 PM

Ive been exploring possible advancements for MMORPGs (with much more complex/detailed game representation on client).   One feature is game Asset data being frequently/continously streamed to the Client    (not just new objects, but terrain updates as almost the entire world is changeable with very little static map data any more  -- this is not just deformable scenery but also a continuing addition of new assets flavors to the game as time goes on - magnitudes more than MMORPGs have these days).

The client would have some huge File dictionaries/encylopedias so that once an instance of an object flavor is transfered it doesnt require a repeat  (and the scheme also has alot of hierarchical templating used to define objects which can cut down alot of the data required to be sent)).

Issues I can think of :

- The Volume of data to transfer that can happen in rushes as the player POV moves  into a much changed location or one with previously unseen object flavors.   Effect upon ongoing gameplay when the gameflow has to share with a background asset data stream  (how much is too much)

- Need for placeholder data to handle the delay while the client 'catches up

- Preloading prioritization (to attempt to preload needed data before it is seeable)  types of objects and distance (LOD considerations)

- In general a much larger number of objects in the clients simulation (keeping the game processing busy)

- Server loads (alot more data traffic to all clients)

- Dictionary/Encyclopedia lookups and update decisions  -- the client has to determine from the server whether there has been an update for an asset flavor (besides NEW asset flavors there are also improvement/fix updates that can happen)

- possibility of offline/background to background generic dictionary updates  (if the immediate-need data flow is Idle then send through changed/new data to get likely future-needed data into the client well ahead of time.)

- Platform considerations.   Limited disk archiving might require discarding some previously transfered data (keeping whats most relevant to the players situation and unfortunately having to reload some data as it is needed.

- Detail level control scheme - need a way to limit/tune this whole system for targeted hardware having significantly less capability (MMORPGs tend to have a wider range of customer target - heck,  tablets continue to advance)

- Higher average Internet throughputs

- SSD to speed the disk end of archiving/caching client data

- the usual CPU/Memory improvement for general increases of performance (additional cores etc..)

- heavy use of a Templatuing scheme with alot of reuse which can help shrink the data that is required to be sent fo any particular object instance

All of this is something for the future - where things MIGHT go  (current state of MMORPGs if not fossilized, have actually degenerated)

---

--------------------------------------------Ratings are Opinion, not Fact

### #2nfactorial  Members

Posted 15 January 2014 - 07:04 AM

Not sure what you're asking really, is this for your own engine? You seem to imply (based on the start of your post "Ive been exploring possible advancements for MMORPGs") that these are not things MMORPGs do. But just about any MMORPG out there performs (and has solved) all the things you have listed, and the items listed are not really suitable for small scale discussion.

n!

Edited by nfactorial, 15 January 2014 - 07:05 AM.

### #3wodinoneeye  Members

Posted 15 January 2014 - 09:32 AM

Not sure what you're asking really, is this for your own engine? You seem to imply (based on the start of your post "Ive been exploring possible advancements for MMORPGs") that these are not things MMORPGs do. But just about any MMORPG out there performs (and has solved) all the things you have listed, and the items listed are not really suitable for small scale discussion.

n!

This is talking about a magnitude of increase asset update traffic, and sending asset defining data (meshes/textures/animations/soundeffects/etc..) WHILE the game is running (NOT some in between splash screen 'level' data loading, nor UIDs of textures/objects that are already in a static dictiionary on the client as the player meets other players/house and such changeable data).

Im talking about constant sets of new data on Server that then never could previously exist on the client, and is too much data to simply statically upload in data patches the way MMORPG games do now.   (the terrain is also now all subdivided objects so the content there is also a maginiture of more dynamic data -- the static baked level is basically gone.

I did some minor work on my own 'engine' years ago but that was more testing chunking on-the-fly procedural generation of terrain between server and client and AI nodes ).

--------------------------------------------Ratings are Opinion, not Fact

### #4nfactorial  Members

Posted 15 January 2014 - 12:42 PM

WHILE the game is running (NOT some in between splash screen 'level' data loading,

Yes, I understand that is what you meant. The streaming technology you mention is fairly common among games these days.

What isn't so common is streaming data over the network, but that is more a data and bandwidth issue rather than a software implementation problem. Large amounts of data over the network means your bandwidth requirement goes up by an order of magnitude and thus reduces the size of your target audience, which is the main reason it isn't done rather than any technical limitation.

n!

### #5AgentC  Members

Posted 15 January 2014 - 01:08 PM

You could take a look at the Second Life client (which is open source C++) for some inspiration. The world consists entirely of user-created content, so there is in general no preinstalled world assets, and data including textures are streamed to the client on the fly, then cached for later access. Whether it's a particularly good or performant implementation, is another matter.

Every time you add a boolean member variable, God kills a kitten. Every time you create a Manager class, God kills a kitten. Every time you create a Singleton...

### #6wodinoneeye  Members

Posted 20 January 2014 - 10:02 PM

You could take a look at the Second Life client (which is open source C++) for some inspiration. The world consists entirely of user-created content, so there is in general no preinstalled world assets, and data including textures are streamed to the client on the fly, then cached for later access. Whether it's a particularly good or performant implementation, is another matter.

I will have to look at that.  I havent seen Second Life since they got rid of the gambling (quite a while ago)

Edited by wodinoneeye, 20 January 2014 - 10:02 PM.

--------------------------------------------Ratings are Opinion, not Fact

### #7Sirisian  Members

Posted 22 January 2014 - 11:20 PM

You can solve this fairly easily by using a CDN. Upload the files to a CDN like Akamai or Rackspace then simply send the url to your clients. Done.

### #8hplus0603  Moderators

Posted 23 January 2014 - 03:55 PM

Sirisian, I think the problem the OP is trying to solve is that the users may modify the environment, and thus static CDN caching won't necessarily work very well.

And, even with a CDN, if the user needs to download more data to see an area than the time it takes to travel there times the user's bandwidth, it won't work, so careful attention to level of detail and environment encoding methods is important.

And finally, when you say "upload," I think you're thinking about a hosting service like S3, which is not a CDN. A CDN works like a web cache, through DNS re-direct, where they will redirect asset requests to a particular domain that you own, to their servers "close" to the user. if the servers don't have a copy, they will let the request through to your main servers to get the data, and likely cache a copy locally for some amount of time to serve further requests for the same data. There is no "uploading" involved, and it doesn't do anything to help users whose bandwidth do not allow downloading enough data "just in time."

enum Bool { True, False, FileNotFound };

### #9Sirisian  Members

Posted 23 January 2014 - 09:12 PM

I was imagining something similar to second life. Stuff is changeable but it doesn't change often. (Unless all the terrain changes every frame?) I think you're greatly underestimating modern content delivery networks also. They can be changed rather frequently and don't require fallback due to their distributed nature. As fast as you can upload the change and transfer the URLs to the clients is all it would take. I use Rackspace (which uses Akami's servers) for something similar at my job to quickly send data to hundreds of clients. I was talking to one of the Guild Wars 2 developers who uses Akami for their 500 MB patching every week. Allows them to hand the files off and distributes them to their clients.

Essentially this gets rid of a few key issues of ensuring low latency data transfer of content to clients. Especially for content that hasn't changed. It greatly cuts down on the processing required by the servers and switches the priority of the server to simply determining what to tell the client to load. With a CDN you can even upload LOD models. You would still have the option to send quick delta compressed changes to a client for terrain changes local to them, but for large blocks of the terrain a CDN would be ideal.

I think one of the hardest issues if one were to use a CDN type approach would be with:

Effect upon ongoing gameplay when the gameflow has to share with a background asset data stream  (how much is too much)

You'd be dealing with a TCP stream that would need to be throttled to not overwhelm the server stream. The client would need to perform this step ideally after they're told what to download.

The idea of streaming data comes up a lot and could probably use similar algorithms to what's used in live streaming from a CDN except instead of variable bit rate codecs it just throttles the stream or switches content in the download queue for lower LOD if it can't keep up.

### #10hplus0603  Moderators

Posted 24 January 2014 - 10:48 AM

Essentially this gets rid of a few key issues of ensuring low latency data transfer of content to clients. Especially for content that hasn't changed.

I think we disagree on what "low latency" means. I think two seconds is pretty high latency for game-affecting change propagation.

Also, you don't "Upload" stuff to Akamai. The easiest way to propagate a new file through Akamai (or any other CDN) is to generate a new URL for it. When the CDN gets a cache miss for this new URL, they will slurp the data from your servers, and subsequent requests will likely be served through the CDN. However, you cannot easily change the data that lives at a certain URL; you have to wait for HTTP caching headers to expire, or make an explicit call to flush the resource to the CDN, and that call may have significant latency (5-50 minutes depending on a lot of factors including which CDN.)

So, if the latency of occsaional HTTP downloads is just fine, then make each block of data available through HTTP. Make the URL of that block be something like http://yourserver.com/blocks/<sha256-of-block-data>. Then you can know that a particular version of a block will be uniquely cached (the chance of accidental SHA-256 collision is effectively zero.) Then, you have to come up with a way to let the clients know that "the block for location (15, 30) is called ad987a987987bbaa987234" -- you can't use (15, 30) as the name/address/URL, because that would require invalidating caches when the content changes.

FWIW, we typically serve 3-4 times as much bandwidth out of Akamai than we do out of our own data center. I don't think we publish our hardware specifics, but we have multiple uplinks, each in the 1-10 Gbps range. For example, if we push 2 Gbps out of our data center, we'd be pushing 6-8 Gbps out of Akamai (which it fills on an as-needed, cache-missed basis.) Now, given that a lot of our data is service data, not statically cacheable data, the actual cache hit ratio is higher than the 75-80% you'd expect from just comparing those two numbers.

Edited by hplus0603, 24 January 2014 - 10:51 AM.

enum Bool { True, False, FileNotFound };

Posted 29 January 2014 - 05:01 PM

### #12wodinoneeye  Members

Posted 08 February 2014 - 02:08 AM

Streaming what part of the game?  And are there issues for some players with delays caused by this?  Any noticable LOD when you run fast towards a previously never preloaded area?

Mainly characters or is this for terrain also (which would be something relatively static you usually could preload ... but then WOW likley has a rather large World Map  or have they added some procedurally generated terrain?)

I remember in Lord of the Ring Online (years ago) you would walk out into some area many player congregated in  (they had long previously segregated the banks to seperate closed  'areas' because of their heavy usage/population)  and it would often take longer than a minute for all the players present to actually pop in to visibility (because all the different clothing/equipment on the avatars and later animated textures they added for those, and I think that data was just the GUIDs not the raw data itself).  And there dId not seem to be any simpler placeholder made visible before the player was fully renderable. (no LOD mechanism).

Edited by wodinoneeye, 08 February 2014 - 02:48 AM.

--------------------------------------------Ratings are Opinion, not Fact

### #13wodinoneeye  Members

Posted 08 February 2014 - 02:33 AM

I guess I need to clarify the data traffic Im talking about :

The level of constant (in-game streamed) data download to the players client is likely to vastly exceed anything done so far.   It would also be in small frequently changing subsets which comes directly from the dynamic data of the game's terrain management servers (with areas of seamless server boundries) - so static download servers cant really apply as the mix-n-match data is different for each player and for THAT point in time (and depends on what the player's client has already cached)

The game would constantly have new assets.  The idea is to have major player creation of assets to multiply new content by several magnitudes.   Along with that is objectifying the entire terrain and the props that usually  these days are static in 'baked levels'.    Players actions can (and will)  add/subtract/modify the terrain and props at any time and new recently published assets can suddenly be there the next time you pass thru.

This would already be using alot of data compression (use of hierachical templates for common 'inherited' data) and the data dictionary on the client end, BUT  would be for ALL asset types  including more unique bulky ones like : textures, body animations, video streams, sound effects, object meshes which the templating only helps with a little.

A appreciable amount of Procedural Generation would be part of this MMORPG and some terrain/scenario building might be done ON the client (so the quite small procedures/seed parameters need only be sent)

One issue - just sending the client fine grained information about what assets (their type and identity) ARE present (where the player is currently or moving towards) BEFORE the client then can check  'do I have a cached copy of this' (and what can I substitute temporarily)  to then need to issue a request to the Servers for pulling missing data.

So thats why I'm trying to get a handle on the effects/problems with that much data flow (or how much internet data flow has to improve to make it viable)   Consider if one client gets this elevated amount, what does that mean for the servers handling thousands of such players simultaneously....

Edited by wodinoneeye, 08 February 2014 - 02:58 AM.

--------------------------------------------Ratings are Opinion, not Fact

### #14hplus0603  Moderators

Posted 08 February 2014 - 11:02 AM

So thats why I'm trying to get a handle on the effects/problems with that much data flow

You will be excluding some segment of players who don't have enough networking to download the experience.

Also, you will have to figure out how to pay for all this, in dollars. You will need both servers, and internet bandwidth, and if the data changes often, traditional CDN caching is unlikely to be very effective.

Good-quality bandwidth in well-connected data centers typically costs about $10 per megabit per second (order of magnitude -- details matter.) Let's say that your game needs 10 Mbps per player. This means that you'll be paying$100 per month per simultaneous online player of your game. How many simultaneous online players do you need? 1,000? If so, the bandwidth bill for your first month will be $100k. enum Bool { True, False, FileNotFound }; ### #15wodinoneeye Members Posted 08 February 2014 - 12:17 PM The game data itself with lots of stuff going on, but thats mostly object IDs/basic data, triggered animations/events and 'surge-y' asset downloads... The 'new' downloads when moving into new or modified areas again will vary depending on how common the resident objects are and how much may have changed. Some of the streamed assets are large even when well compressed but eventually it gets caught up and that load diminishes. Some good LOD prioritizing mechanism obviously will help (alot of these assets you dont see full detail til you are right ontop of them, but complementary to that is to have them ready by the time you ARE ontop of them. And as this is a MMORPG I can/could slide a bit on 'perfect' detailing of game objects. I'm not even quite sure what typical DSL is these days (effective rate), but I do expect improvements by the time some game like this would actually be operational (Im also counting on SSD s being more common) I'll have to re-investigate sizing of typical objects at dif LODs to better judge what might be needed. Even within one 'prop' - say a 'chair', there are the broken pieces meshes, the X different damage/dirt effects textures/decals/mesh patches, and similar other atypical 'states' which may not have to be downloaded immediately for first full view (and all the LOD simplifications for several steps of distant viewing likely being quite alot less data). Its possible there will be more than a little material texture sharing that will cut need for unique textures down (theres also the issue of how much data can fit in the targeted GPU memory too) I would hope the required thru-put might average closer to 1Mbps instead of 10Mbps (how surge-y ). All the other costs of running a MMORPG arent insignificant (and THIS design idea is trying to cut out alot of the company asset creation costs to try to bring down expenses as well as make niche games more possible (after all, a 'chair' in one genre's game mechanics is largely the same data as in another if you modularize it sufficiently). Big budget for all the easy to (player) use tools -- that though will be the killer/delayer for this idea. Oh and If I ran such a thing there would be NO Free-To-Play (is the jury out yet as to whether that income model actually is working for most games who have it??) - I would have a solid monthly game charge like the old model (~$20-25 a month ??)

Edited by wodinoneeye, 08 February 2014 - 12:24 PM.

--------------------------------------------Ratings are Opinion, not Fact

### #16hplus0603  Moderators

Posted 08 February 2014 - 04:33 PM

I would hope the required thru-put might average closer to 1Mbps

You said what you're doing is on a scale that hasn't been seen before. Second Life streams user-generated content, and uses at least that amount of bandwidth to do it. Whether user-generated streaming content will be good enough for the masses (and attract enough users who are good at generating content!) is a real challenge.

OK, so if you support 1,000 simultaneous online players, your first bandwidth bill will be $10k. I hope you have the budget needed to get that off the ground! (Also: If you're buying less bandwidth, there's a point where the per-megabit cost will be pretty high because of fixed overhead costs.) enum Bool { True, False, FileNotFound }; ### #17wodinoneeye Members Posted 09 February 2014 - 07:09 AM Dont worry you dont have to "read me the Riot Act" they do to so many here of these forums for even thinking that 'I" am going to make a MMORPG etc etc... Its more trying to work out of what I 'hope' might be down the line for us in the future. Like I said first a better more interactive game, second lots of new imaginitive content, third a business model that works (including smaller niche games using same system possibly load leveling the same server/network resources). Look at MMORPGs and see how pathetic they are (actually degenerating from games like Ultima Online) and Im looking to see what might get them out of the 'gonna be the next WOW' rut the companies and their investors are stuck in. Likely it would require a game engine company collaboration... even more risky. Take the cost of a Triple A game ($100 mil these days?) and double it because of the tools to enable practical player asset creation (up to and including mission scenarios, behavior scripting, etc...)    What company is going to gamble on that??  ... even if what they build up could be reused on their next 6 games (some of it even is just superior tools to significantly cut costs even if there is no revolutionary  'player creation' involved).

SO the typical downstream load of Second life is about 1Mbps? Continuous?   How do they afford to pay the rate (just the data xport) you've mentioned?   I expect the game mechanism I'm thinking of may be very 'bursty/surge-y'  depending on player activities/behaviors/patterns  (if they sit still alot it falls off alot)   so fluctuating between that 10Mbps and well below 1Mbps may be the actual case with the low end typical.

A fixed monthly charge without them endlessly selling to you within the game (poisoning it in my opinion).  How much could you charge for a superior MMORPG game (or game system with multiple genres/themes to select from) ?

Still I expect years off the thru-put costs will drop, we might have a few more adaquate open source tools to leverage, and maybe some game company who sees the trap they are in and wants to push into the next generation of gaming ???

Edited by wodinoneeye, 09 February 2014 - 07:10 AM.

--------------------------------------------Ratings are Opinion, not Fact

### #18hplus0603  Moderators

Posted 09 February 2014 - 11:42 AM

Second Life can afford the networking because their business model is leasing servers that run Second Life software. The actual players pay for the server/bandwidth with the "land fees."

Another game that does live editing/streaming is EverQuest Next / Landmark, although that's not shipped yet.
enum Bool { True, False, FileNotFound };

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.