hi all,

how do commercial/professional game engines handle resources like images, fonts, sounds during release/package of the game?

i mean ive been writing graphics applications including mini games but the images for example, are loaded straight up from directory.

like (pseduo code):

imageloader.load("/images/sprites.png");

i know this is not safe as people have access to resource and they can "hack" it buy changing the resource, etc.

ive got books related to engine making, but most if not all of them do not have a chapter nor a section about this.

how do professionals do it?

1. Zip the entire resource and unzip it somewhere?

please enlighten me.

Depends on the products.

Usually during development it is as you described, loading the files directly.

Then later in development, usually more efficient methods are built. This is usually done by switching file formats to match what is friendly in memory.

Consider that the big cost of loading files isn't the disk transfer, it is figuring out how to store and process the data in the file. A seemingly simple file like a large XML document can be an enormous monster when it comes to memory allocation inside a program; naive programmers will build enormous data trees requiring untold thousands of memory allocations to construct the representation.

Many games avoid this by packing the resources to exactly the format they will be needed on the destination machine. The game can load it all to a large block of memory, fix some pointers to specific locations within the data file, and everything automatically works.

Other times they use data formats that are better suited to the systems. You mention PNG files in your example. While PNG files are great for downloading images on the web since they are tightly compressed, they are terrible for games because of how they expand. There are various texture file formats that can be loaded directly to the card and don't need to be decoded or processed. Their compression ratio is not nearly as tight as PNG, but you can directly pass the file to the graphics card and use it instantly, so it works well.

Many games use tools similar to PhysFS, using a compressed file structure for the final game to save space -- basically a zip file -- but also loading resource files if they exist on disk. This allows developers to work mostly from assets as they will be shipped to customers but still do development with individual files as needed.

i know this is not safe as people have access to resource and they can "hack" it buy changing the resource, etc.

That is generally not a big concern. Yes there are cheaters out there who will replace wall textures with alpha-enabled textures so they can see through them. People will replace enemy textures with bright red or bright orange colors so they can be seen easily. While games can attempt to take countermeasures against it the difficulty and cost are extremely high, and it will always be a "cat and mouse" style game of attackers finding a way thorough, then developers finding and stopping it, the attackers finding another way through, and developers finding and stopping it, repeating until the product is no longer profitable.

Better to just know up front that cheaters are going to cheat and spend your time and energy into creating the best game you can.

Depends on the products.

Usually during development it is as you described, loading the files directly.

Then later in development, usually more efficient methods are built. This is usually done by switching file formats to match what is friendly in memory.

Consider that the big cost of loading files isn't the disk transfer, it is figuring out how to store and process the data in the file. A seemingly simple file like a large XML document can be an enormous monster when it comes to memory allocation inside a program; naive programmers will build enormous data trees requiring untold thousands of memory allocations to construct the representation.

Many games avoid this by packing the resources to exactly the format they will be needed on the destination machine. The game can load it all to a large block of memory, fix some pointers to specific locations within the data file, and everything automatically works.

Other times they use data formats that are better suited to the systems. You mention PNG files in your example. While PNG files are great for downloading images on the web since they are tightly compressed, they are terrible for games because of how they expand. There are various texture file formats that can be loaded directly to the card and don't need to be decoded or processed. Their compression ratio is not nearly as tight as PNG, but you can directly pass the file to the graphics card and use it instantly, so it works well.

Many games use tools similar to PhysFS, using a compressed file structure for the final game to save space -- basically a zip file -- but also loading resource files if they exist on disk. This allows developers to work mostly from assets as they will be shipped to customers but still do development with individual files as needed.

i know this is not safe as people have access to resource and they can "hack" it buy changing the resource, etc.

That is generally not a big concern. Yes there are cheaters out there who will replace wall textures with alpha-enabled textures so they can see through them. People will replace enemy textures with bright red or bright orange colors so they can be seen easily. While games can attempt to take countermeasures against it the difficulty and cost are extremely high, and it will always be a "cat and mouse" style game of attackers finding a way thorough, then developers finding and stopping it, the attackers finding another way through, and developers finding and stopping it, repeating until the product is no longer profitable.

Better to just know up front that cheaters are going to cheat and spend your time and energy into creating the best game you can.

thank you for this detailed reply, i will also try PhysFS.

For my resources, I wrote a custom archive builder.

The tool works like this.

When given a target folder, the target folder becomes the archives name. This also gets marked as the apparent root for the entire file.

So... files that are read as,

/Archive/data.txt

become

/data.txt

in both code, and archiving. Really the file name only matters when determining load order.

So... as folders get deeper and deeper, the tool use that as a large string, but it also tries to keep data in a sensible order.

"/images/characters/texture1.targa" is the entire string that is hashed.

All hashed strings are sorted and stored in a header in the archive. This header marks the location of the actual file so the file system can immediately jump to what it needs.
Once the header log is made, the file is compressed and appended to the end of the data segment with some more data.

So now you have the registry information about that file.

This is a little bit that tells you the file type, the compression algorithm, the compressed size, and the real size of it. Directly after that is going to be the actual data for that file.

This process gets repeated to the very last file.



Soooo... now it comes to loading.

The engine loads in the headers for all game archives, and stores them in a big ass array. Any overlapping names are overwritten by a stacking load order, where the last is the most visible.

When the game needs data, it uses a string like "/textures/characters/human.targa". The game will immediately hash this on the fly using the same exact algorithm and seed as my tool, and look it up in it's table. Locate the data, with the proper file name. And load it from there. This hash can also be predetermined, but it will take some work to set that up.

It reads the compression size first, and allocates the required space in the scratch space. It then reads the loaded size and allocates it in a more permanent residence. The compressed data is loaded into the scratch, and then uncompressed into the resident space.

i've done resource files, custom formats, and standard formats in the past (IE pretty much all the ways you can).

as frob says, don't worry about folks modding / hacking assets.

i've come to the conclusion that loose standard file formats that load fast enough are the mod friendly (and thus user friendly) way to go.

all-in-one resource files can be slightly faster, as you only open one file, but its probably not noticeable on today's hardware.

all-in-one resource files are less mod friendly, and more work to implement.

binary memory image files are nice and fast:

int array[max]

FILE f

fwrite_binary(f,array,sizeof(int)*max)

custom formats for speed/size when standard formats won't do. in my current project i need to switch from text to binary .x files. but blender doesn't export binary. haven't looked for a converter solution yet.

As he said, there's generally no point in worring about people hacking the data files now a days. It's going to happen, so usually developers do not bother with this. If for some reason you absolutely must prevent people from hacking things due to it effecting the experience of other players, than you can do what Blizzard did.

Most of their scripts are on server only. Animation, particles, etc are handled by the client.