I was recently thinking about it, and went and threw together a mechanism for the server part of an engine to push files over to the client.

basically, it is rigged up in such a way that client-side file-open events are intercepted, and if the file is in the cache, this version will be used (otherwise it will fall back into checking in the engine's VFS).

as-is, the pushed files are not actually saved to the HDD, but are kept in RAM.

the idea could possibly be that of moving away from the client-end having its own permanent resource files (basically, the resource file that come with the game), and instead possibly working differently:

A, all resource data (textures / sounds / models / ...) are always streamed to the client (at the cost of potentially wasting bandwidth for network games, *1);

B, the client implements a persistent cache, with some sort of "precache" mechanism used to keep the cached files up to date (either a dynamically generated list, or possibly some sort of "general manifest");

C, leaving most resource-data as permanent files, with separate per-server streamed or cached files;

...

if a persistent cache were used, likely it would need to be kept server specific.

cache files would likely keep track of the file-sizes and a checksum, and if the "precache" server event contains files which aren't in the client-side cache, or the sizes or checksums are different, it will send a message back to the server, and the server will push over any of the files.

while this would save bandwidth (by not resending unchanged files), it would add some complexity to the handling of caches.

*1: basically, if every time the client would connect to the server, the server would send over any textures or sound-effects used by the world, which could potentially be a fairly big chunk of data (the worry is if whenever a client connects, it has to be sent 50 or 100MB or more of resource data, which could suck...).

ideally I guess, we would want the client to remember any files previously given to it, so that they don't have to be sent every time the client connects.

another minor uncertainty is how to best handle the case where the client has an out-of-date file:

simply invalidate the stale file, and wait for the new one to become available before loading the resource;

use the old file, and maybe "invalidate" the loaded copy somehow once the new one is downloaded, causing it to be reloaded (currently, no such invalidation mechanism exists);

alternatively, all file downloading/... is handled prior to the client being put into the world (*2).

*2: so, handshake process works like:

client connects to server socket (and does low-level handshake);

client sends, a "(connect)" message to server;

server sends:

general server information;

a dump of the world contents (voxel terrain and map geometry, lists of lights and entities, ...);

a list of precached files (new).

client validates (upon seing a precache list) that all its files are up to date:

if true, sends back an "(connect_ok)";

if false, it sends back a list of stale files ("(precache_stale files...)").

server handles client messages:

if it sees a "(precache_stale)" message, it pushes any relevant files;

if it sees "(connect_ok)", it spawns the clients' player-entity and dumps them in the world.

handshake process until client sends "(connect_ok)";

(at this point, all resources are assumed up-to-date / ...)

client loads any data and begins rendering;

communication now consists primarily of "(impulse)" and "(wdelta)" messages (client input impulses, and world-deltas).

probably even with persistent caches, it could still make sense for the server to be able explicitly push data (say, because the data is session-specific or should never be saved to the HDD, which could probably be specially marked somehow, *3).

maybe "general pushes" ("(push_file)") should not be done during gameplay, but only during the initial handshake.

note that there are special "push" messages, namely "(push_sound)" and "(push_image)", which more explicitly patch over already loaded sounds or textures.

*3: say, ".volatile/...", where the prefix would basically tell the client "don't save this to disk!".

for the permanent-files scenario, this mostly works under the premise that all the resource data would need to be included for single-player to work anyways, so one may as well just use it also for network gameplay (these files would not be streamed by the server, leaving streaming mostly for server-local resources, rather than general game data).

nevermind, I still lack any really "good" way to handle client/server GUI stuff, but alas...

thoughts / comments?...

This already exists in games like MMORPGs (EQ2 ? even WoW to some degree), where the client streams the data when needed, thought most will be saved to the HDD to speed up loading the next time. As long as HDD space is much more available than bandwidth, caching your data on HDD is the way to go. But this depends on a lot of factors, e.g. goolge maps loads most of its maps on-the-fly, which is suiteable for this usecase. But loading a whole bunch of models/textures/sounds etc. will consume a lot of bandwidth and time, and the latter will scare off lot of people if overdone.

When it comes down to implementation, I would sugguest to leave it actually out of the game-client. Use some kind of launcher and standard protocol (ftp) to handle the data stream before launching the game. The issues with non standard protocols are, that they could results in unnecessary problems with the file-server backend. Eventually, once the game data and community reaches a certain threshold, you will need some quite powerful fileserver backend (or use an existing cloud service), which will most likely only support some standard protocols.

If you have a trivially small set of assets, sure, why not.

If you have any significant volume of data, streaming the assets every time the game relaunches is going to make a tremendously bad user experience.

This already exists in games like MMORPGs (EQ2 ? even WoW to some degree), where the client streams the data when needed, thought most will be saved to the HDD to speed up loading the next time. As long as HDD space is much more available than bandwidth, caching your data on HDD is the way to go. But this depends on a lot of factors, e.g. goolge maps loads most of its maps on-the-fly, which is suiteable for this usecase. But loading a whole bunch of models/textures/sounds etc. will consume a lot of bandwidth and time, and the latter will scare off lot of people if overdone.

When it comes down to implementation, I would sugguest to leave it actually out of the game-client. Use some kind of launcher and standard protocol (ftp) to handle the data stream before launching the game. The issues with non standard protocols are, that they could results in unnecessary problems with the file-server backend. Eventually, once the game data and community reaches a certain threshold, you will need some quite powerful fileserver backend (or use an existing cloud service), which will most likely only support some standard protocols.

this game isn't really for an MMO, so most servers would likely be player-run (sort of like is typical with Quake3, Doom3, or Minecraft servers).

a lot of player-run servers would likely have some amount of customized assets (say, customized for whatever worlds they are hosting on them).

for home users, FTP and HTTP servers would be a bit more of a pain to set up, as well as some ISPs blocking incoming FTP and HTTP for end-user accounts (forcing use of non-standard ports). handling these via the games' protocol would make it easier to get through a firewall (since if the main game-port gets through the firewall, file-transfer is available as well).

but, yes, file retrieval via HTTP has been considered as well.

I ended up just adding it directly to the game-protocol, and it is currently using similar mechanisms to that used for passing voxel terrain data, where basically the game protocol has special mechanisms to pass largish byte arrays.

a game-launcher would not be ideal, as it assumes a centralized repository of resource data, rather than the resource data being specific to each server the player might try to connect to, which will not usually be known until in-game, when the player tries to connect to a server.

If you have a trivially small set of assets, sure, why not.

If you have any significant volume of data, streaming the assets every time the game relaunches is going to make a tremendously bad user experience.

yeah, that is the problem.

as-is, a client connecting to a server will typically be given several MB of voxel-data (for the locally visible parts of the world), so the question then partly becomes how much more all the textures / ... would add to this. this is potentially a lot of data though.

so yeah, probably will need some sort of persistent file-cache.

most likely such a cache could be held in a directory where all the files are mostly stored using hashed-names, probably with per-server manifest lists.

I don't know why can't you just go the way ID did for pretty much the whole time up to Doom 3: they just asked for a resource and if it didn't resolve, the client would pull it from the server. AFAIK there was no hashing involved and there was no "cache". Files were just dumped to the base directory.

Although simple, it was effective and sufficient in practice. Once we put an checksum on top of it, it becomes fairly robust in my opinion.

I suggest against using a per-server cache: in my experience back when gaming was a more prominent part of my life, a lot of servers had common assets.

I don't know why can't you just go the way ID did for pretty much the whole time up to Doom 3: they just asked for a resource and if it didn't resolve, the client would pull it from the server. AFAIK there was no hashing involved and there was no "cache". Files were just dumped to the base directory.

Although simple, it was effective and sufficient in practice. Once we put an checksum on top of it, it becomes fairly robust in my opinion.

I suggest against using a per-server cache: in my experience back when gaming was a more prominent part of my life, a lot of servers had common assets.

there is a chance though that different servers could have assets with the same names and different contents, or possibly that multiple versions of a given asset could exist, in which case we probably want to use "the right version" for that server.

though the possibility of commonly shared assets is interesting. it could be possible to partly unify the cache, and reuse a file from a different server if path+size+checksum happen to match (at which point it can be assumed to be the same file). in this case, rather than the cache being divided for each server, it could remember per-server which version of a given file is being used.

the specifics of how the files are stored on disk is (or in RAM) pretty well hidden from most of the code, so most of the engine code doesn't really need to know/care how the files are stored (the VFS layer makes them all look pretty much like normal files, so from the POV of the client code, it is like the local filesystem contents would vary from one server to another).

the need for a "precache" mechanism for downloading though is mostly because otherwise there is no good way to know that a file needs to be downloaded (there are no hooks into the VFS for this, and a lot of the engine code uses "probing" to load things, *1). IIRC, it was also the "precache" mechanism which was used to trigger downloading in the idTech engines.

*1: basically, where a partial file path is given, and the loader-code will try out each possible location and file-extension it knows of for that type of resource until it either finds something or gives up when it doesn't find anything.

or such...

Don't identify assets by a name. Identify them by the combination of a name and a checksum or simple one-way hash (MD5 is popular for this).

Don't identify assets by a name. Identify them by the combination of a name and a checksum or simple one-way hash (MD5 is popular for this).

yeah.

in my cache, I switched to primarily identifying them via a combination of name+size+checksum.

if all of these match, it is assumed to be the same file, otherwise it is different.

the code for on-disk caching has since been written, but the logic for checking against a server-provided manifest and requesting files for download has not yet been written.

the current cache logic will skip writing any files to disk which have path-names beginning with '.', which for now will be assumed to be volatile and kept solely in-RAM.