Jump to content

  • Log In with Google      Sign In   
  • Create Account

How to avoid slow loading problem in games


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
38 replies to this topic

#1 warnexus   Prime Members   -  Reputation: 1438

Like
0Likes
Like

Posted 01 February 2013 - 12:12 AM

As you know one of the problems that plague a game is slow loading time. It breaks immersion. IMHO, this game does it the most noticealy. The game is Spectral Souls on the PSP. I made a 50 second video recording my character exiting a store and entering back in.

 

 

Exits starts at 0:02 and ends at 0:20 - 18 seconds total. A game that says disc access while having a black screen can't be good either.

 

Entering the store starts at 0:23-0:37 - 14 seconds total. So I guess the game might have the data of the previous area you entered stored temporarily somewhere and that saves you 4 seconds? 

 

I do the same test again.

 

Exiting the store starts at 0:38-0:47-8 seconds total. Wow now it is much less time to load!

 

Did the programmers not test the algorithms for execution time?

 

How does a programmer prevent this from happening?

 

I'm currently writing my own game in Java and I certainly learned a lot about this game loading time. Although I only recorded entering and exiting the store, the same applies the same way the game loads animation files of a spell or executes an algorithm to control monster's behavior. Each dialogue screen also needs 5-8 seconds before the dialogue appears.

 

I would not want to implement this type of loading in my game. This is the only game I have come across that does things this bad. I'm pretty sure every gamer prefer smooth transition in loading.



Sponsor:

#2 Ashaman73   Crossbones+   -  Reputation: 7512

Like
0Likes
Like

Posted 01 February 2013 - 12:31 AM

First off, games do often more then just loading data when loading a level first time. They need to prepare the level, unpack data etc. Then they cache it and next time you load it, loading is much faster, this happens often on systems where the data carrier (DVD/cardridge) has only limited memory amounts.

 

The best solution seems to be streaming, that is, the level is not completely loaded at the beginning, often memory expensive data like textures are left out. After the game starts, the rest is loaded, aka streamed, part for part while the game is running.


Edited by Ashaman73, 01 February 2013 - 12:31 AM.


#3 Khatharr   Crossbones+   -  Reputation: 3003

Like
6Likes
Like

Posted 01 February 2013 - 01:21 AM

The PSP has very small RAM and the seek time for UMDs is absolutely horrid. The best thing to do is to use a fast compression/decompression library (because the decompress is faster than reading the full data from the disc would be) and have the data compressed in such a way that it can be decompressed directly into the memory it belongs in without the need for post processing. It's a step beyond serialization, but it's not too hard to do with the flyweight pattern and some creative disc arrangement. 18 seconds of load time is ridiculous even for the PSP. Although the RAM is limited it sounds like either your initial 18 seconds was a bad disc seek or else the game is using a resource cache and ends up not needing to load as much when you immediately re-enter an area.

 

Another method of reducing load times is to use formats that take up less space. On the PSP the CLUT-based 16bpp color mode can be pretty compact even for raw fullscreen bitmaps and can still look super-sexy. The gpu can render directly from this type, so....

 

Basically reducing load times is just like any other optimization: profile and correct. Find the bottleneck and work on it until it's not a bottleneck anymore, then move on to the next bottleneck.


void hurrrrrrrr() {__asm sub [ebp+4],5;}

There are ten kinds of people in this world: those who understand binary and those who don't.

#4 KulSeran   Members   -  Reputation: 2539

Like
7Likes
Like

Posted 01 February 2013 - 03:25 AM

+1 to Khatharr's assessment.  Fast heavy compression cuts down the time you waste waiting on the disk. And in-memory formats save you from doing too much processing at load time.

imho the big pitfalls programmers run into for loading data are:

1) Parsing text files. It's simple, and convenient. It often offers benefits like being able to read your data in any text editor and often being able see your edits on-the-fly in game. Pre-baked binary formats take a bit more effort at edit time because you need a tool to pack text->binary before the game can reload it, and changing the binary structure often means re-building all files of a type. But moving over to a binary format saves space, and wasted compute time parsing the text.

 

2) Sparse data trees laid out as individual files.  It's really nice to have your data broken up as a "index" and "items". But then you end up reading dozens of files to pull together all the information you need. Consider a model. You have the model verts, the material definition, some textures, a shader. Overall this is maybe 7 files for one model and you're jumping all over the disk to get this information.  No matter how you order the loading, seeking around the files will waste time. It's preferable to just pack everything for the level ahead of time into a more optimal layout in a single file. Reading the single file is fast and simple. You can thusly load all the data first, preferably packed up by their usage pattern, and then link it up in memory after the monolithic pack files are loaded. Now you load just a few files (all model/level mesh data, all textures, player mesh data, player textures, player sounds, level sounds) and spend your time jumping around memory to link things up. Much faster.

 

3) Allowing things to be allocated in memory during the load process. You should be able to pre-process all your data to know exactly how many of each thing you have and how large each thing is and reserve all the memory up front.  Something like "std::vector::push_back()" has amortized O(1) insert, but it will waste a TONNE of time re-sizing and copying if you don't call "std::vector::reserve(totalObjectCount)" first. It's the same idea for all the data the game is loading.


Edited by KulSeran, 01 February 2013 - 03:28 AM.


#5 warnexus   Prime Members   -  Reputation: 1438

Like
0Likes
Like

Posted 01 February 2013 - 06:00 PM

+1 to Khatharr's assessment.  Fast heavy compression cuts down the time you waste waiting on the disk. And in-memory formats save you from doing too much processing at load time.

imho the big pitfalls programmers run into for loading data are:

1) Parsing text files. It's simple, and convenient. It often offers benefits like being able to read your data in any text editor and often being able see your edits on-the-fly in game. Pre-baked binary formats take a bit more effort at edit time because you need a tool to pack text->binary before the game can reload it, and changing the binary structure often means re-building all files of a type. But moving over to a binary format saves space, and wasted compute time parsing the text.

 

2) Sparse data trees laid out as individual files.  It's really nice to have your data broken up as a "index" and "items". But then you end up reading dozens of files to pull together all the information you need. Consider a model. You have the model verts, the material definition, some textures, a shader. Overall this is maybe 7 files for one model and you're jumping all over the disk to get this information.  No matter how you order the loading, seeking around the files will waste time. It's preferable to just pack everything for the level ahead of time into a more optimal layout in a single file. Reading the single file is fast and simple. You can thusly load all the data first, preferably packed up by their usage pattern, and then link it up in memory after the monolithic pack files are loaded. Now you load just a few files (all model/level mesh data, all textures, player mesh data, player textures, player sounds, level sounds) and spend your time jumping around memory to link things up. Much faster.

 

3) Allowing things to be allocated in memory during the load process. You should be able to pre-process all your data to know exactly how many of each thing you have and how large each thing is and reserve all the memory up front.  Something like "std::vector::push_back()" has amortized O(1) insert, but it will waste a TONNE of time re-sizing and copying if you don't call "std::vector::reserve(totalObjectCount)" first. It's the same idea for all the data the game is loading.

Thanks for the knowledge. This is very useful information! rolleyes.gif My other question would be I'm sure the programmers know about the slow loading times but why not fix it?


Edited by warnexus, 01 February 2013 - 06:00 PM.


#6 Khatharr   Crossbones+   -  Reputation: 3003

Like
0Likes
Like

Posted 02 February 2013 - 04:47 AM

warnexus, on 01 Feb 2013 - 16:08, said:
...I'm sure the programmers know about the slow loading times but why not fix it?

Maybe ran out of time.
Maybe ran out of money.
Maybe ran out of ****s to give. (Sometimes this happens.)
Maybe just got bushwhacked by the company bureaucracy.
void hurrrrrrrr() {__asm sub [ebp+4],5;}

There are ten kinds of people in this world: those who understand binary and those who don't.

#7 BGB   Crossbones+   -  Reputation: 1554

Like
3Likes
Like

Posted 02 February 2013 - 11:17 AM

good loading times are hard.

 

 

for example, in my case (on a PC), loading textures is a good part of the startup time, and a lot of this due to resampling the (generally small) number of non-power-of-2 textures to be power-of-2 sizes. this is then followed by the inner loops for doing the inverse-filtering for PNG files (more so, the cycle-eating evilness of the Paeth filter, which wouldn't be as bad, except that this tends to be one of the most generally effective, and thus most-used, filters in PNG).

 

I sometimes also wish that PNG also had a simpler "A+B-C" linear filter, which can often compete well with Paeth, but, more notably, is much cheaper (it could improve performance by potentially dilluting the number of times the encoder picks the Paeth filter). granted, along similar lines, one can also wish that PNG filtered per-block rather than per-scanline, ... but alas. (nevermind all the pros and cons of using custom image formats for textures...).

 

note that even as such, decoding such a texture may still be faster than loading a simpler uncompressed texture would be (such as BMP or TGA), due mostly to the time it requires to read files from the HDD.

 

 

luckily, one can usually cache textures.

unluckily, you still have to load them the first time they are seen.

 

there is a trick though, namely to have alternate lower-resolution and high-resolution versions of textures.

initially, only the low-resolution versions are loaded, and then any high-resolution and extended-component textures (normal-maps, ...) are streamed in during play.

 

clever streaming = using a thread. in my case (poor-man's streaming), it often means using a timer, meaning that a certain number of milliseconds can be used doing whatever (if too much time has gone by, we abort and wait until later). granted, for longer operations, this lazy option can still result in chopiness (mostly due to the potentially high number of milliseconds loading a texture can require, potentially going somewhat over the millisecond budget).

 

if graphical settings are set low, also the high-resolution versions may also be skipped entirely.

 

 

sometimes, loading may be hindered by other things, such as a few common offenders:

parsing text files;

linear lookups.

 

linear lookups are extra bad, as they can turn loading from an an O(n) operation into an O(n^2) operation.

 

IME, linear lookups have more often been a bigger problem than traditional parsing tasks, like reading off tokens or decoding numbers.

 

matching strings (such as for command-names) has sometimes been an issue, but shares a common solution with that of the linear lookup problem: hashing.

 

say, for example:

read in token (or read in line and split into tokens);

use a hash-based lookup, mapping the token to a command-ID number or similar;

"switch()".

 

 

a bigger problem with text formats is more often not actually the parsing, but rather reading them in from disk.

 

basically, for reading lots of small files, the OS's filesystem is often your enemy.

 

potentially better is, when possible, to bundle them into an archive, such as a ZIP, then fetch the files from this.

if implemented well, reading the contents from a ZIP archive can actually outperform reading them via normal file-IO (both due to bundling, and also reducing total disk IO via storing the data in a deflated form).

a downside though is that there are planty of braindamaged and stupid ways to handle this as well.

 

(if not implemented stupidly, it is possible to get good random access speeds to a ZIP archive with 100k+ files, but if implemented stupidly, so help you...).

 

granted, due to the way ZIP is designed, the above may still require the initial up-front cost of reading-in the central directory and transforming it into a more efficient directory tree structure (for example, large flat lists of directory-entry nodes, internally linked into a heirarchical tree structure, such as to allow the directory tree to be more efficiently "descended into", like in the OS filesystem).

 

during development, a disadvantage of ZIP though is that it can't be readily accessed by the OS or by "normal" apps (such as graphics editors, ...), so is more something for "final deployment". often these will be given a special extension (such as "pk" or "pk3" or similar) to reduce the likelihood of a naive user extracting them.

 

 

for some other specialized use cases, I am using a loosely WAD-based format. it is also linked into a heirarchy, albeit less efficiently (via simply linking to the parent directory entry) mostly to save space (ideally, we also want a 'next' link, but for A vs B, the parent-link won vs the next-link for the use-case). like ZIP, contents are usually deflated. this can avoid some of the up-front cost (but, with their own costs).

 

 

but, even with everything, getting everything loaded up in a few seconds or less isn't really an easy task (with modern expectations for content).

loading time has to come somewhere, either at engine startup, at world loading, or during gameplay.

 

typically, compromises are made.

 

 

not to say though that some games don't just have bad loading code...

 

 

> void hurrrrrrrr() {__asm sub [ebp+4],5;}

LOLz

(quickly going nowhere fast...).



#8 Khatharr   Crossbones+   -  Reputation: 3003

Like
0Likes
Like

Posted 02 February 2013 - 02:50 PM

> void hurrrrrrrr() {__asm sub [ebp+4],5;}

LOLz

(quickly going nowhere fast...).

 

<3


void hurrrrrrrr() {__asm sub [ebp+4],5;}

There are ten kinds of people in this world: those who understand binary and those who don't.

#9 MrDaaark   Members   -  Reputation: 3555

Like
6Likes
Like

Posted 02 February 2013 - 03:51 PM

good loading times are hard.
 
for example, in my case (on a PC), loading textures is a good part of the startup time, and a lot of this due to resampling the (generally small) number of non-power-of-2 textures to be power-of-2 sizes. this is then followed by the inner loops for doing the inverse-filtering for PNG files (more so, the cycle-eating evilness of the Paeth filter, which wouldn't be as bad, except that this tends to be one of the most generally effective, and thus most-used, filters in PNG).
 
I sometimes also wish that PNG also had a simpler "A+B-C" linear filter, which can often compete well with Paeth, but, more notably, is much cheaper (it could improve performance by potentially dilluting the number of times the encoder picks the Paeth filter). granted, along similar lines, one can also wish that PNG filtered per-block rather than per-scanline, ... but alas. (nevermind all the pros and cons of using custom image formats for textures...).
 
note that even as such, decoding such a texture may still be faster than loading a simpler uncompressed texture would be (such as BMP or TGA), due mostly to the time it requires to read files from the HDD.
 
luckily, one can usually cache textures.
unluckily, you still have to load them the first time they are seen.

Why would you have any data in non optimal dimensions, and why would you even bother loading any of those formats? This stuff should be done once, and only once. Write a tool that goes over your assets folder and does any needed conversions.

Take all your files, and convert them to proper resolution (better yet, beat your artist with a blunt object until he makes proper power of 2 images. Images look bad when you change their aspect ratio), and then turn them into compressed DDS files. You're writing a game, not an image editor, it doesn't need to know about PNG or TGA or anything else.

Then when your game runs, dump your compressed DDS directly into VRAM. You don't need to check their dimensions. You don't need to uncompress or convert anything. Just dump it directly. DDS also handles mip levels, which covers your next paragraph.

When the game is running, everything should already be ready to go. Nothing should ever have to be converted or processed on the fly. Keep a good copy of everything for editing purposes, but make sure it's all converted when you build your game. Do a simple date comparison, if any asset is newer than the last converted one, convert it then.

#10 BGB   Crossbones+   -  Reputation: 1554

Like
-1Likes
Like

Posted 02 February 2013 - 07:05 PM

good loading times are hard.
 
for example, in my case (on a PC), loading textures is a good part of the startup time, and a lot of this due to resampling the (generally small) number of non-power-of-2 textures to be power-of-2 sizes. this is then followed by the inner loops for doing the inverse-filtering for PNG files (more so, the cycle-eating evilness of the Paeth filter, which wouldn't be as bad, except that this tends to be one of the most generally effective, and thus most-used, filters in PNG).
 
I sometimes also wish that PNG also had a simpler "A+B-C" linear filter, which can often compete well with Paeth, but, more notably, is much cheaper (it could improve performance by potentially dilluting the number of times the encoder picks the Paeth filter). granted, along similar lines, one can also wish that PNG filtered per-block rather than per-scanline, ... but alas. (nevermind all the pros and cons of using custom image formats for textures...).
 
note that even as such, decoding such a texture may still be faster than loading a simpler uncompressed texture would be (such as BMP or TGA), due mostly to the time it requires to read files from the HDD.
 
luckily, one can usually cache textures.
unluckily, you still have to load them the first time they are seen.

Why would you have any data in non optimal dimensions, and why would you even bother loading any of those formats? This stuff should be done once, and only once. Write a tool that goes over your assets folder and does any needed conversions.

Take all your files, and convert them to proper resolution (better yet, beat your artist with a blunt object until he makes proper power of 2 images. Images look bad when you change their aspect ratio), and then turn them into compressed DDS files. You're writing a game, not an image editor, it doesn't need to know about PNG or TGA or anything else.

Then when your game runs, dump your compressed DDS directly into VRAM. You don't need to check their dimensions. You don't need to uncompress or convert anything. Just dump it directly. DDS also handles mip levels, which covers your next paragraph.

When the game is running, everything should already be ready to go. Nothing should ever have to be converted or processed on the fly. Keep a good copy of everything for editing purposes, but make sure it's all converted when you build your game. Do a simple date comparison, if any asset is newer than the last converted one, convert it then.

 

 

power-of-2 is good, but mandating power of 2 isn't always itself optimal from a content-creation POV. better IMO to accept whatever, and resample it as needed, but generally as a matter of policy try to keep everything power-of-2 (to avoid resampling).

 

a lot of the ugliness caused by resampling can be reduced by using bicubic interpolation for the texture resampler, but granted, doing this is slow (somewhat slower than bilinear or nearest interpolation).

 

 

the problem with DDS is that it chews though a lot more HDD space than PNG or JPEG, and the potential for faster loading is largely hindered by the additional disk IO required (still better than BMP or TGA, which is just wasting HDD space). like, some saved clock-cycles doesn't really offset a slow HDD all that well (as well as still leaving download times longer for end-users, ...).

 

JPEG tends to gives the smallest files, but being lossy tends to result in lower image quality, and (in its common form) does not support alpha blending.

PNG is more of a compromise.

 

granted, I have a custom format (roughly based on JPEG, but with more features), but it has its own drawbacks (mostly that it is non-standard).

 

 

the Paeth issue more has to do with decoding PNG images, where essentially one ends up with several conditionals per pixel component (3 or 4 times per pixel):

given it is needed to do this about 1M times for a 512x512 image, and conditionals are fairly slow, this isn't free (especially since the branch-predictor can't really accurately predict them).

a direct linear predictor (a+b-c) usually does almost as well as Paeth, but is cheaper (but, sadly, not supported by PNG).

an encoder which then avoids Paeth ends up having to use a generally much less effective filter (resulting in bigger compressed images).

 

ironically, this issue is sufficiently bad that it manages to (almost single handedly) make decoding PNG images currently slower than decoding JPEG images.

 

but, alas...



#11 Endurion   Crossbones+   -  Reputation: 3575

Like
2Likes
Like

Posted 03 February 2013 - 12:45 AM

As Daark hinted at, being nice to all kind of mangled formats/dimensions is useful during production, but once you hit the finish line, store the stuff in the exact format you need in memory.

 

Simple block reading is the best you can do for loading times.

 

Depending on the target system you may have to make compromises though (file size vs. loading speed).


Fruny: Ftagn! Ia! Ia! std::time_put_byname! Mglui naflftagn std::codecvt eY'ha-nthlei!,char,mbstate_t>

#12 MrDaaark   Members   -  Reputation: 3555

Like
0Likes
Like

Posted 03 February 2013 - 02:27 AM

cr88192, this is all complete nonsense. Please don't get defensive. Instead of trying to be "right", learn from your mistakes and become better at your craft.

HDD space is a near infinite resource these days, and this thread is about in-game loading times. No one is going to miss the few extra bytes a DDS may use over another format.

Every optimization has a drawback. You always use more of something else to achieve your goal of using less of whatever the bottleneck is.

Just like mipmaps take up 33% more space per texture. It's not that expensive, and it solves the problem very well.

We used to use more memory to create look up tables to avoid expensive computations. Now we often do computations over again to avoid cache look-up misses.

Disc based games often game data in ways that a whole section can be streamed in at once, instead of storing it in a way that takes up less disc space. It doesn't hurt anything to use up all the space, and the loading times can improve dramatically.

GPUs are built to use DDS textures. They are supported "as is", both to save space, and to improve efficiency when they are being passed around. You get quicker texture loading (remember, this is a quicker loading thread!), less VRAM usage, and better bandwidth usage all for free.

power-of-2 is good, but mandating power of 2 isn't always itself optimal from a content-creation POV. better IMO to accept whatever, and resample it as needed, but generally as a matter of policy try to keep everything power-of-2 (to avoid resampling).

Every piece of art ever created had restrictions on it's dimensions. Be it 512x512, 8.5x11 (A1 legal), the size of a poster board, the size of a wall in a cave, etc... If size of 2 doesn't fit the object to be textured, then you work on it in whatever dimensions you want WITHIN a power of 2 canvas.

The art can be whatever dimensions it wants. The canvas MUST be power of 2.

a lot of the ugliness caused by resampling can be reduced by using bicubic interpolation for the texture resampler, but granted, doing this is slow (somewhat slower than bilinear or nearest interpolation).

Changing the aspect ratio of an image destroys it in many ways. The dimensions of a piece, which are very important to establish it's look and feel become distorted. Wide things become thin, thin things become wide, curves get straightened out, or go from being short, to long and vise-versa. This isn't a programming issue, so you can't program your way out of it. It doesn't matter what filter or technique you use. Changing the aspect ratio of an image completely changes it.

#13 C0lumbo   Crossbones+   -  Reputation: 2274

Like
1Likes
Like

Posted 03 February 2013 - 06:55 AM

As someone mentioned the UMD has particularly poor seek times, additionally if you don't read any data from the UMD for awhile, the disk stops spinning (to save battery life), when you start reading again there is a significant delay as it spins back up to speed. It's possible the developers of the game you cite made a decision to let the UMD spin down to save your battery life, OTOH, it could just be inefficient.

 

Here is a decent article about how to organise the data you load so that there is a minimal amount of CPU work to be done on load - http://www.gamasutra.com/view/feature/132376/delicious_data_baking.php (the TLDR version is, 1. Load your data in place, 2. Fixup the pointers. i.e. it won't be anything new to anyone who has optimised loading time or the CPU footprint of streaming loading on a console).



#14 EWClay   Members   -  Reputation: 659

Like
-1Likes
Like

Posted 03 February 2013 - 10:53 AM

Changing the aspect ratio of an image destroys it in many ways. The dimensions of a piece, which are very important to establish it's look and feel become distorted. Wide things become thin, thin things become wide, curves get straightened out, or go from being short, to long and vise-versa. This isn't a programming issue, so you can't program your way out of it. It doesn't matter what filter or technique you use. Changing the aspect ratio of an image completely changes it.


The aspect ratio of a texture in the game depends on what it is mapped on to. Changing the size of the texture won't affect that, unless the mapping is generated dynamically using the texture size.

Not to take away from the main point that artists should be authoring textures in power of two sizes in the first place.

#15 BGB   Crossbones+   -  Reputation: 1554

Like
0Likes
Like

Posted 03 February 2013 - 10:54 AM

cr88192, this is all complete nonsense. Please don't get defensive. Instead of trying to be "right", learn from your mistakes and become better at your craft.

I am writing mostly from personal experience, which tends to be that HDD's aren't very fast.

while there is often plenty of space on an HDD, its read/write speeds are not necessarily all that fast.

HDD space is a near infinite resource these days, and this thread is about in-game loading times. No one is going to miss the few extra bytes a DDS may use over another format.

Every optimization has a drawback. You always use more of something else to achieve your goal of using less of whatever the bottleneck is.

Just like mipmaps take up 33% more space per texture. It's not that expensive, and it solves the problem very well.

We used to use more memory to create look up tables to avoid expensive computations. Now we often do computations over again to avoid cache look-up misses.

Disc based games often game data in ways that a whole section can be streamed in at once, instead of storing it in a way that takes up less disc space. It doesn't hurt anything to use up all the space, and the loading times can improve dramatically.

the problem is, IME, loading times are often largely IO-bound, and often a DDS texture will take about 2x-4x as much space as a PNG (though, a DDS is still about 1/4 to 1/3 the size of a raw BMP).


if it takes 2.6s for the disk-time to read these files, but at 1/2 the size takes 1.3s, unless decoding requires more than 1.3s, it is still a net saving.

hence, a strategy of both organizing data for streaming, and try to make the data to be streamed smaller.

hence why a person may also use *deflate* to make disk-IO faster.
decoding deflated data isn't free, but still much faster than waiting for the data to be read-in otherwise.


the issue is where one reaches a point of "diminishing returns", namely where decoding the data would outweigh the IO-time savings.

#16 BGB   Crossbones+   -  Reputation: 1554

Like
0Likes
Like

Posted 03 February 2013 - 11:44 AM


Changing the aspect ratio of an image destroys it in many ways. The dimensions of a piece, which are very important to establish it's look and feel become distorted. Wide things become thin, thin things become wide, curves get straightened out, or go from being short, to long and vise-versa. This isn't a programming issue, so you can't program your way out of it. It doesn't matter what filter or technique you use. Changing the aspect ratio of an image completely changes it.

The aspect ratio of a texture in the game depends on what it is mapped on to. Changing the size of the texture won't affect that, unless the mapping is generated dynamically using the texture size.

Not to take away from the main point that artists should be authoring textures in power of two sizes in the first place.


yes, agreed.
using power-of-2 is best, just no particular reason it should be mandatory for the loader, except in special cases (for example, my code for streaming video-streams into textures does mandate a power-of-2 size, and a specific set of codecs).

in most of the cases, where the size is already correct, then no resampling is needed.

warning about non-power-of-2 textures may be a good idea though, such that any which are found can be hopefully resized to the correct size outside the game (be it complaining to the artist or whatever, so that they can go and fix it).


as for aspect/etc:
typically, only the images' "virtual size" is really needed for calculating texture coordinates, which may be largely independent of its actual resolution.

if a non-power-of-2 texture is encountered, it will be resampled internally, but its virtual size will remain unchanged.
the main issue is the possibility of visual distortion introduced by resampling, which isn't usually too much of an issue (though, yes, it is still better if resampling isn't needed in the first place).

#17 EWClay   Members   -  Reputation: 659

Like
1Likes
Like

Posted 03 February 2013 - 12:42 PM

Regarding texture loading:

In most cases you should be using one of the block compressed formats, as they use less GPU memory and save bandwidth. But compression is slow, so it wouldn't make sense to decompress from something like jpeg then re-compress, even though jpeg is smaller. Instead, block compress the texture first, offline, then compress that (e.g. with Zlib). This will give around another 50% saving, and the only work to do on loading is decompression, direct to the final format.

There are specialised lossy compression schemes for block compressed textures that can do even better, getting close to jpeg levels.

#18 cardinal   Members   -  Reputation: 885

Like
2Likes
Like

Posted 03 February 2013 - 01:30 PM

It must be nice living in a world where HDDs have essentially an unlimited amount of space. At work I have 3 HDDs (one of which is a terabyte) and am constantly struggling to find space anytime a lot of assets are added. Anyways, on to something actually relevant... In my personal experience with the PSP (I worked on several PSP games years ago) the disk seek times are slow (as others have said). The main trick in dealing with this is to layout your data on the disk in a way that reduces these seek times. To do that you want data that is loaded together to be together on the disk so the UMD doesn't have to seek back and forth to read data. Another trick is to duplicate data in multiple places on the disk if it is loaded in different places. This allows you to seek less to read the same data. The problem with that approach is that UMD capacity is limited, so it only works if you have the space to play around with. Some games are easier to handle than others (for instance games that load on a per-level basis would be easier to manage than something that stream loads or loads when entering new areas (provided the areas are non-linear).

#19 BGB   Crossbones+   -  Reputation: 1554

Like
0Likes
Like

Posted 03 February 2013 - 03:08 PM

Regarding texture loading:

In most cases you should be using one of the block compressed formats, as they use less GPU memory and save bandwidth. But compression is slow, so it wouldn't make sense to decompress from something like jpeg then re-compress, even though jpeg is smaller. Instead, block compress the texture first, offline, then compress that (e.g. with Zlib). This will give around another 50% saving, and the only work to do on loading is decompression, direct to the final format.

There are specialised lossy compression schemes for block compressed textures that can do even better, getting close to jpeg levels.

yeah, DDS+deflate is possible I guess, and would come by default if a DDS is packaged up in a ZIP.


I was able to shave off a bit little more from the loading time, mostly by manually resampling some of the non-power-of-2 textures, and also thinking about it and finding a trick to eliminate the conditionals from the Paeth filter regarding PNG loading (realizing that there was a pure integer-math solution). current engine startup time is ~3s, and getting the world loaded up is around 15s.

most of the loading-related time-usage (according to profiler) is still within "ntoskrnl.exe" though.
other observations are that in both loading events, the HD light turns solid red for a few short bursts.

resource monitor: during loading, the engines read-IO spikes at around 50MB/s.
during gameplay has continued IO of around 1MB/s to the voxel-terrain (region files), but most of the CPU usage is related to voxel-related stuff (followed by "ntoskrnl.exe" and "nvoglv32.dll").

#20 phantom   Moderators   -  Reputation: 7278

Like
5Likes
Like

Posted 03 February 2013 - 05:22 PM

being nice to all kind of mangled formats/dimensions is useful during production, but once you hit the finish line, store the stuff in the exact format you need in memory.

 

We don't even allow that.

 

EVERY resource which is loaded, during every stage of development, is a processed one. All images are converted first to DDS then to our custom format (basically a smaller header with a platform specific payload attached which is in the hardware format required) so they can be directly streamed in as fast as possible.

 

The only difference to development vs final build is that for the final build the various packaged files are consolidated into larger volumes which are compressed using zlib.

 

There is no good reason to be dicking about at runtime with custom formats and if you aren't giving the GPU DXTn/BCn compressed data to work with then you are Doing It Wrong™ and a GOOD DXTn/BCn compresseor can take quite some time to run so if you are trying to run time compress to any decent degree then there is no way you'll get fast loading times.






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS