Jump to content
  • Advertisement
Sign in to follow this  
cebugdev

Handling game resources

This topic is 737 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

hi all,

 

how do commercial/professional game engines handle resources like images, fonts, sounds during release/package of the game?

i mean ive been writing graphics applications including mini games but the images for example, are loaded straight up from directory.

like (pseduo code): 

imageloader.load("/images/sprites.png");

 

 

i know this is not safe as people have access to resource and they can "hack" it buy changing the resource, etc.

ive got books related to engine making, but most if not all of them do not have a chapter nor a section about this.

 

how do professionals do it?

1. Zip the entire resource and unzip it somewhere?

 

 

please enlighten me.

Share this post


Link to post
Share on other sites
Advertisement

Depends on the products.

 

 

 

Usually during development it is as you described, loading the files directly. 

 

Then later in development, usually more efficient methods are built.  This is usually done by switching file formats to match what is friendly in memory.

 

Consider that the big cost of loading files isn't the disk transfer, it is figuring out how to store and process the data in the file.  A seemingly simple file like a large XML document can be an enormous monster when it comes to memory allocation inside a program; naive programmers will build enormous data trees requiring untold thousands of memory allocations to construct the representation.

 

Many games avoid this by packing the resources to exactly the format they will be needed on the destination machine. The game can load it all to a large block of memory, fix some pointers to specific locations within the data file, and everything automatically works.

 

Other times they use data formats that are better suited to the systems.  You mention PNG files in your example. While PNG files are great for downloading images on the web since they are tightly compressed, they are terrible for games because of how they expand. There are various texture file formats that can be loaded directly to the card and don't need to be decoded or processed. Their compression ratio is not nearly as tight as PNG, but you can directly pass the file to the graphics card and use it instantly, so it works well.

 

Many games use tools similar to PhysFS, using a compressed file structure for the final game to save space -- basically a zip file -- but also loading resource files if they exist on disk. This allows developers to work mostly from assets as they will be shipped to customers but still do development with individual files as needed.

 

i know this is not safe as people have access to resource and they can "hack" it buy changing the resource, etc.

 

That is generally not a big concern.  Yes there are cheaters out there who will replace wall textures with alpha-enabled textures so they can see through them. People will replace enemy textures with bright red or bright orange colors so they can be seen easily. While games can attempt to take countermeasures against it the difficulty and cost are extremely high, and it will always be a "cat and mouse" style game of attackers finding a way thorough, then developers finding and stopping it, the attackers finding another way through, and developers finding and stopping it, repeating until the product is no longer profitable.

 

Better to just know up front that cheaters are going to cheat and spend your time and energy into creating the best game you can. 

Share this post


Link to post
Share on other sites

Depends on the products.

 

 

 

Usually during development it is as you described, loading the files directly. 

 

Then later in development, usually more efficient methods are built.  This is usually done by switching file formats to match what is friendly in memory.

 

Consider that the big cost of loading files isn't the disk transfer, it is figuring out how to store and process the data in the file.  A seemingly simple file like a large XML document can be an enormous monster when it comes to memory allocation inside a program; naive programmers will build enormous data trees requiring untold thousands of memory allocations to construct the representation.

 

Many games avoid this by packing the resources to exactly the format they will be needed on the destination machine. The game can load it all to a large block of memory, fix some pointers to specific locations within the data file, and everything automatically works.

 

Other times they use data formats that are better suited to the systems.  You mention PNG files in your example. While PNG files are great for downloading images on the web since they are tightly compressed, they are terrible for games because of how they expand. There are various texture file formats that can be loaded directly to the card and don't need to be decoded or processed. Their compression ratio is not nearly as tight as PNG, but you can directly pass the file to the graphics card and use it instantly, so it works well.

 

Many games use tools similar to PhysFS, using a compressed file structure for the final game to save space -- basically a zip file -- but also loading resource files if they exist on disk. This allows developers to work mostly from assets as they will be shipped to customers but still do development with individual files as needed.

 

 

 

i know this is not safe as people have access to resource and they can "hack" it buy changing the resource, etc.

 

That is generally not a big concern.  Yes there are cheaters out there who will replace wall textures with alpha-enabled textures so they can see through them. People will replace enemy textures with bright red or bright orange colors so they can be seen easily. While games can attempt to take countermeasures against it the difficulty and cost are extremely high, and it will always be a "cat and mouse" style game of attackers finding a way thorough, then developers finding and stopping it, the attackers finding another way through, and developers finding and stopping it, repeating until the product is no longer profitable.

 

Better to just know up front that cheaters are going to cheat and spend your time and energy into creating the best game you can. 

thank you for this detailed reply, i will also try PhysFS.

Share this post


Link to post
Share on other sites

For my resources, I wrote a custom archive builder.

 

The tool works like this.

When given a target folder, the target folder becomes the archives name. This also gets marked as the apparent root for the entire file.

So... files that are read as,

/Archive/data.txt

 

become

 

/data.txt

 

in both code, and archiving. Really the file name only matters when determining load order.

So... as folders get deeper and deeper, the tool use that as a large string, but it also tries to keep data in a sensible order.

"/images/characters/texture1.targa" is the entire string that is hashed.

All hashed strings are sorted and stored in a header in the archive. This header marks the location of the actual file so the file system can immediately jump to what it needs.
Once the header log is made, the file is compressed and appended to the end of the data segment with some more data.

So now you have the registry information about that file.

This is a little bit that tells you the file type, the compression algorithm, the compressed size, and the real size of it. Directly after that is going to be the actual data for that file.

 

This process gets repeated to the very last file.



Soooo... now it comes to loading.

The engine loads in the headers for all game archives, and stores them in a big ass array. Any overlapping names are overwritten by a stacking load order, where the last is the most visible.

When the game needs data, it uses a string like "/textures/characters/human.targa". The game will immediately hash this on the fly using the same exact algorithm and seed as my tool, and look it up in it's table. Locate the data, with the proper file name. And load it from there. This hash can also be predetermined, but it will take some work to set that up.

It reads the compression size first, and allocates the required space in the scratch space. It then reads the loaded size and allocates it in a more permanent residence. The compressed data is loaded into the scratch, and then uncompressed into the resident space.

Edited by Tangletail

Share this post


Link to post
Share on other sites

i've done resource files, custom formats, and standard formats in the past (IE pretty much all the ways you can).

 

as frob says, don't worry about folks modding / hacking assets.

 

i've come to the conclusion that loose standard file formats that load fast enough are the mod friendly (and thus user friendly) way to go.

 

all-in-one resource files can be slightly faster, as you only open one file, but its probably not noticeable on today's hardware.

 

all-in-one resource files are less mod friendly, and more work to implement.

 

binary memory image files are nice and fast:

 

int array[max]

FILE f

fwrite_binary(f,array,sizeof(int)*max)

 

custom formats for speed/size when standard formats won't do. in my current project i need to switch from text to binary .x files. but blender doesn't export binary. haven't looked for a converter solution yet.

Share this post


Link to post
Share on other sites

i've done resource files, custom formats, and standard formats in the past (IE pretty much all the ways you can).

 

as frob says, don't worry about folks modding / hacking assets.

 

i've come to the conclusion that loose standard file formats that load fast enough are the mod friendly (and thus user friendly) way to go.

 

all-in-one resource files can be slightly faster, as you only open one file, but its probably not noticeable on today's hardware.

 

all-in-one resource files are less mod friendly, and more work to implement.

 

binary memory image files are nice and fast:

 

int array[max]

FILE f

fwrite_binary(f,array,sizeof(int)*max)

 

custom formats for speed/size when standard formats won't do. in my current project i need to switch from text to binary .x files. but blender doesn't export binary. haven't looked for a converter solution yet.

 

As he said, there's generally no point in worring about people hacking the data files now a days. It's going to happen, so usually developers do not bother with this. If for some reason you absolutely must prevent people from hacking things due to it effecting the experience of other players, than you can do what Blizzard did.

Most of their scripts are on server only. Animation, particles, etc are handled by the client.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!