Sign in to follow this  

Loading resources from archive

This topic is 3043 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Soon I'm going to make my game load resources from an archive. Is it better to load all the needed resources in one go or is it just as good to load a file one by one?

Share this post


Link to post
Share on other sites
Quote:
Original post by ill
Soon I'm going to make my game load resources from an archive. Is it better to load all the needed resources in one go or is it just as good to load a file one by one?

Depends on the size of the resources and how/when you're using them. The general rule of thumb is to load resources only when needed, but depending on the needs of your application you may want to have certain resources loaded and ready to go for future operations.

For example, if all or most of your resources are in constant use throughout your game, it would be wise to pull them all at once and have them sitting handy in memory for faster swapping and usage. On the other hand, if you are using only, say, ten of the files at any given point out of a total 100 in the archive, it would be best to leave the unused ones in the archive until needed; having the unused files sitting otherwise in memory is a waste of valuable space.

Be wary of the size of the resources you're pulling in. If all of your resources combined constitute several hundred megabytes or more, then loading all the resources into memory may not be feasible given the limitations of a computer's available RAM.

Another point worth considering is how fast you need your resources. If during play you for some reason need a particular resource immediately, having the resource already loaded into memory will allow that. If it's not crucial to have a certain resource right away--but you know that it will be needed sometime soon--then it may work out best to start fetching the resource in a separate thread process so that the resource can either be completely loaded or at least on its way for when it is actually needed.

Share this post


Link to post
Share on other sites
That all makes sense.

My real question was though, is it a good idea to load a file one by one or as a batch process? Like is it better to load like 10 things in one go or is it alright to load 1 thing at a time? I'm talking about speed of loading. Like say I load 10 items one after another one at a time, or I load all 10 items at once, is the loading considerably faster when it's doing 10 at once?

I could test it but that'll take me a while and I want to start making the resource loader soon. Like I see a lot of resource loading functions for images and stuff where it says, load from file, or load from archive. Do those functions load just that one file from an archive or would it be a better idea for me to extract the needed files into some temporary place on disk in one go and load them that way into the game? Also is it possible to extract files into memory directly and have an image loading function load it from there?

Basically the game is 2D so there aren't going to be massive 3D textures being thrown around.

Share this post


Link to post
Share on other sites
Quote:
My real question was though, is it a good idea to load a file one by one or as a batch process? Like is it better to load like 10 things in one go or is it alright to load 1 thing at a time? I'm talking about speed of loading. Like say I load 10 items one after another one at a time, or I load all 10 items at once, is the loading considerably faster when it's doing 10 at once?


This really depends on your archive format and platform. If you are using an archive that simply is one giant file of all the files you need and are on Windows, you can make use of the Memory Mapping API for efficient file processing that allows you to have instant access to all files and no new allocations needed. In that case, you don't need to "load" anything as you already have the buffer of the file available. If you use compression and encryption though, then you will need to create additional working buffers to process files one at a time in, but that's the tradeoff for a little sense of security.

So, you will usually be loading files one at a time regardless of what setup you have. The concept of loading more than one file at once just isn't applicable since each is an independent operation. If it were a dependent operation, you'd be accessing your files in an O(n) algorithm and that would be a real problem for a lot of files!

If you had multiple threads running and loading files, then that's another story and complexity, but for what you are talking about, loading files one at a time is standard. A lot of games do it that way and I've seen games with tens of thousands of files using it too (mmos)!

Quote:
Like I see a lot of resource loading functions for images and stuff where it says, load from file, or load from archive. Do those functions load just that one file from an archive or would it be a better idea for me to extract the needed files into some temporary place on disk in one go and load them that way into the game?


Saving the file to a temporary place on disk just to load it again would defeat the main purpose of using an archive! You want to keep everything in memory because it's a lot faster than your disk I/O is going to be. The only exception here is if you have a file that's highly compressed and it's too large to decompress nicely into memory so you decompress it in chunks in memory and then write out the chunks into a file to rebuild it. From there, you'd use a a memory mapped file handle to access it rather than loading it into memory.

Unless you have gigabytes of resources or are targeting old computers with less than 512MB of ram, you should not really run into memory problems. Windows will make use of the paging file as needed to which your application has a private virtual 2GB of memory to use. If you are on another OS, well you'll need to read up on the memory management there and it might be a problem.

Quote:
Also is it possible to extract files into memory directly and have an image loading function load it from there?


Yes, how that is setup is up to your image loading functions though. Most image libraries support loading from memory as it is pretty much a necessary feature nowadays. Rather than opening a file and reading the data into a buffer and operating on it, a function is setup to specify the buffer to operate on and that's it. If the loading code is more complex and makes sequential fread/ReadFile calls, you'd just use pointer arithmetic and type casting on the stream instead and you end up with the same result.

The key here is to not over think it or get too crazy. Identify why you want to use an archive first. Look into using an existing library such as PhysFS or the 7z/zip format. A zlib based format is alright too as you can just create one large data file of all the contents of compressed files and store a secondary file that tells you what you need to know to access the files (size, name, offset, crc). Finally, think about how you will handle updates because you surly don't want to have to redistribute the entire archive for a small change in a file or something!

Good luck!

Share this post


Link to post
Share on other sites
Quote:
Original post by ill
That all makes sense.

My real question was though ...

Ack, I answered the wrong question! Looks like Drew gave a good response, though. Sorry if my words were not very helpful.

Good luck with the resource loader!

Share this post


Link to post
Share on other sites

This topic is 3043 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this