Sign in to follow this  
streamer

how to speed up texture loading?

Recommended Posts

streamer    415
Well, I have an annoying problem. In my DX8 game during loading, I must load cc 7000 textures ( or more), and because of game loading after few restarts(debugging or similar)i'm getting a little bit nervous :). Is there any way to speed up this process? This is an isometric game so every animation is handled as set of textures (one character animation is 520 texture long, for example). I was thinking about something like this, but don't know how to do it: - setting up one animation in an large custom file - load custom file in one step - decompress file in memory - load from memory Anyway, is this going to speed up whole process? And how can be done this?

Share this post


Link to post
Share on other sites
Henri    122
Having 7000 ID3DTexture9 objects is not a good idea. Try to batch up all textures belonging to one animation in one texture. Then load that texture and animate the texture coordinates.

I don't really know how you are doing your drawing, but it should be quite simple. That will save you a lot of overhead and make the loading faster.

If you want, you could also try batching up some of the animations in one texture. In general having a big texture (1024*1024) is a lot better than many small ones, as it will save you state changes.

Share this post


Link to post
Share on other sites
ganchmaster    134
Hopefully these are pretty small textures!

In any case, one convenient way to batch these textures together would be to use volume (3D) textures. Then the animation is easy too, as it's just a matter of animating the third texture coordinate. You can save volume textures as .dds files and load them as a single chunk.

Share this post


Link to post
Share on other sites
streamer    415
Well yaeh. I tried that, but that solution gave me much more headache, than simple texture loading. That it is because the textures are not fixed size, and also they aren't all pow^2, so in this solution I have pixels lost, the textures are not so sharp etc...
And then if animation is 128x128 big, this means that in texture of 1024x1024 size I have only 64 animations (in my character animations of 570 frames this is not enough).
And last thing, if I know well some older cards cannot load textures of that size..

Share this post


Link to post
Share on other sites
streamer    415
Quote:
Original post by ganchmaster
Hopefully these are pretty small textures!

In any case, one convenient way to batch these textures together would be to use volume (3D) textures. Then the animation is easy too, as it's just a matter of animating the third texture coordinate. You can save volume textures as .dds files and load them as a single chunk.


yes. these are pretty small textures, but I have also big ones.
Can I have in the volume texture, textures of variable sizes? And volume textures aren't depended on video card, I mean all video cards can open volume textures?

Share this post


Link to post
Share on other sites
Jiia    592
I think your setup is fine, as long as the video card supports the image sizes in memory. It would be bad if it tried to pad the images to make them square or powers of 2.

To speed it up, you might want to consider compressing the textures on the hard drive, and storing all related textures in one file. Load each file in, uncompress, then transfer the data to individual texture surfaces. This should speed up loading time considerably, depending on the type of images you use and the type of compression you try.

You could just save the textures as PNG (decent compression), then create a simple tool to shove all of one animation's frames into a single file (just store each file's byte data in sequence). In your engine, load the whole file into memory at once, then use the DirectX texture LoadFromMemory functions to store each texture.

Just some ideas.

Share this post


Link to post
Share on other sites
streamer    415
Quote:
Original post by Jiia

You could just save the textures as PNG (decent compression), then create a simple tool to shove all of one animation's frames into a single file (just store each file's byte data in sequence). In your engine, load the whole file into memory at once, then use the DirectX texture LoadFromMemory functions to store each texture.
.


My textures are already in PNG format, and I come to similar solution of my problem.
But DXLoadTetxureFromMemory will speed up the whole process? I dont want to link 7000 textures if there are no speedup improvement.

And how about the BIG game-developing companies? They how solve this kind of problems?

Share this post


Link to post
Share on other sites
Jiia    592
Quote:
Original post by streamer
But DXLoadTetxureFromMemory will speed up the whole process? I dont want to link 7000 textures if there are no speedup improvement.

There's an easy way to test it. Create a file sized by about the size of one your images. Open it, allocate it's size, read it from file, close the file, then delete the memory, 7000 times. Then create a single file 7000 times larger. Open it, allocate, read, close, then delete, once. I would expect about 300% speed increase, but that's just a random guess. I might be totally wrong. Easy to test, though.

Share this post


Link to post
Share on other sites
streamer    415
Quote:
Original post by Jiia
Quote:
Original post by streamer
But DXLoadTetxureFromMemory will speed up the whole process? I dont want to link 7000 textures if there are no speedup improvement.

There's an easy way to test it. Create a file sized by about the size of one your images. Open it, allocate it's size, read it from file, close the file, then delete the memory, 7000 times. Then create a single file 7000 times larger. Open it, allocate, read, close, then delete, once. I would expect about 300% speed increase, but that's just a random guess. I might be totally wrong. Easy to test, though.


Yeah you were right :) It is much more faster. I just don't know how will DXLoadTextureFromMemory behave in my program. But this seams to be a right point. Again, lot of changes in code. I think this will be one more sleepless night.
Thanx

Share this post


Link to post
Share on other sites
Jiia    592
Quote:
Original post by streamer
I just don't know how will DXLoadTextureFromMemory behave in my program. But this seams to be a right point. Again, lot of changes in code. I think this will be one more sleepless night

It's incredibly simple. Instead of passing a filename, you just pass the memory pointer and byte size. The hard part would be combining all of your files as you update art [smile]

Share this post


Link to post
Share on other sites
streamer    415
Quote:
Original post by Jiia
Quote:
Original post by streamer
I just don't know how will DXLoadTextureFromMemory behave in my program. But this seams to be a right point. Again, lot of changes in code. I think this will be one more sleepless night

It's incredibly simple. Instead of passing a filename, you just pass the memory pointer and byte size. The hard part would be combining all of your files as you update art [smile]


ineresting [smile].
I rewroted a texture load proc like this:


D3DXIMAGE_INFO imageInfo;

HRESULT hr = E_FAIL;

char* buffer;
int fh;

if( ( fh = _open( buf, _O_RDONLY | _O_BINARY ) ) != -1 )
{
// file length
long fl = _filelength( fh );

buffer = new char[fl];

// read binary data
if ( _read( fh, buffer, fl ) > 0)
{
// read from memory
hr = D3DXCreateTextureFromFileInMemoryEx(m_pd3dDevice,buffer,fl,0,0,0,0,
D3DFMT_UNKNOWN, D3DPOOL_MANAGED, D3DX_DEFAULT,
D3DX_DEFAULT, dwCol, &imageInfo, NULL, &m_pTexture);


// save image dimensions
m_imageWidth=imageInfo.Width;
m_imageHeight=imageInfo.Height;
}


}


and seems to be a a little bit faster to load texure as binary file, and then load it as texture from memory [rolleyes]

Share this post


Link to post
Share on other sites
Jiia    592
You mean it's faster to do this with just a single file? That is interesting. Are you running in release mode? DirectX libraries may be doing a lot of in-between calls when opening files, which may slow the routine down in debug mode. Or they may just be taking a lot of precautions. It's hard to tell.

By the way, out of curiousity, how are you saving your images? Are you exporting them from a renderer? How do you get them saved with individual sizes? Do you have an automatic cropping tool?

Share this post


Link to post
Share on other sites
streamer    415
Quote:
Original post by Jiia
You mean it's faster to do this with just a single file? That is interesting. Are you running in release mode? DirectX libraries may be doing a lot of in-between calls when opening files, which may slow the routine down in debug mode. Or they may just be taking a lot of precautions. It's hard to tell.

By the way, out of curiousity, how are you saving your images? Are you exporting them from a renderer? How do you get them saved with individual sizes? Do you have an automatic cropping tool?


Yeah it is faster with single file and I'm running it in debug mode. Release mode is just faster.
I don't think I understand the second part. Screenshoots from game I grab with CTRL+PrtScr and Paste it in Paint[smile]. If you are thinking about game animation well that is harder part. I render 3D model with 3DMax onto isometric viewport. Output file have several properties which you can define in file output properties. In this section I defined a custom color for backround(sprites and everything in game is alpha-blended)and output size. To get a proper character size inside a texture, in Max you can load an picture as background. Then you need just to set up proper disteance of camera and voala 3DMax is doing the rest. Same thing for objects.

Share this post


Link to post
Share on other sites
Ninebit    122
If I get you right, you wan't to SPEED THINGS UP. To do that you must sacrifice memory. Get rid of the image compression. Uncompression takes time. Next: do you need all texturefiles at once? If not, load them as you go on. Load the ones you need at the start and load the others in a thread in the background. Or better yet, if you know that you will only need a certain texture at a certain point in the game (say animating a boss monster or whatever), load when you need it and then dump it.

Storing all files in one big binary/archive file will do one thing for you: getting rid of 6999 file open and close. The rest is pretty much the same.

Load from memory: Well you gotta gett it into memory first don't you?

a 520 frame animation?? thats 22 secs of video, or I would say 64 secs of ingame animation (or whatever you have there). Did you ever think about streaming video instead?

Putting all animations in one file could be good, but then again: isn't that just a video file? I like to collect game data (like textures) in larger files though, but not because it loads faster...

Share this post


Link to post
Share on other sites
Jiia    592
Quote:
Original post by Ninebit
Get rid of the image compression. Uncompression takes time.

I know this seems like a simple fact, but in most cases, it's not. Reading more data from the hard drive (or worse, the CD) is usually much slower than reading a (smaller) compressed version of that data and uncompressing it in memory. This can also be tested by comparing loading speeds of PNG vs BMP, which PNG passes with flying colors on my system.

Share this post


Link to post
Share on other sites
streamer    415
Quote:
Original post by Ninebit
If I get you right, you wan't to SPEED THINGS UP. To do that you must sacrifice memory. Get rid of the image compression. Uncompression takes time. Next: do you need all texturefiles at once? If not, load them as you go on. Load the ones you need at the start and load the others in a thread in the background. Or better yet, if you know that you will only need a certain texture at a certain point in the game (say animating a boss monster or whatever), load when you need it and then dump it.


Unfortunetly I can't. I need them all. It is mostly background graphic. Animations is another thing. And textures are in PNG format which have very good compression ratio and also simple decompress routine. And it is fast.

Quote:

Load from memory: Well you gotta gett it into memory first don't you?


Yes I need to get it into memory first. But! I think that DirectX is messing up with something when DXCreateTextureFromFile is called...anyway it slows a loading sequence...and as I said it seems to be a faster to load a texture as binary file in memory and then create a texture from pointer. I don't understand too why it is so. It is just a little bit faster.

Quote:

a 520 frame animation?? thats 22 secs of video, or I would say 64 secs of ingame animation (or whatever you have there). Did you ever think about streaming video instead?


If you played Diablo or any other RPG you would notice that character animations is created from texture sequences. Streaming video is used for internet video streaming, not for games.

Quote:

Putting all animations in one file could be good, but then again: isn't that just a video file? I like to collect game data (like textures) in larger files though, but not because it loads faster...


But the point of this topic is that I'm writing an MMORPG game and loading sequence is too long, and THAT is annoying thing. When I need to restart a game several times because I looking up for some bug, and when I wait 20-30sec per restart then I start to think that maybe a faster game start is better idea.

Share this post


Link to post
Share on other sites
Dakiar    124
It sounds a little strange to need all the 7000 files at once? are you sure you really need that many?, and if the problem is that debugging takes too long, I'd suggest replacing many of the files with the same file while debugging? or are you trying to debug the animations / backgrounds ? it probably be easier to do it in some other way, then to run the game each time.

Share this post


Link to post
Share on other sites
Jiia    592
Quote:
and seems to be a a little bit faster to load texure as binary file, and then load it as texture from memory [rolleyes]

I think I may understand why this is true. The file-opening routine probably feeds a memory pointer (pointer to the hard drive) to the same function that you're using to load from memory. If so, the file-opening version would end up reading bit by bit from the hard drive, instead of reading the entire file at once. Reading a big chunk at once should be a bit faster.

Not that any of this is important. Just thought I would share my guess.

Share this post


Link to post
Share on other sites
rKallmeyer    396
It looks like you are passing 0 for Mip Map field. Do you really need to generate a full mip-map chain for every texture? That right there could be killing your performance. Try chaning it to 1 to see what kind of speed boost you get...

Share this post


Link to post
Share on other sites
Guimo    463
Hi streamer,
In your code:

a. Trying to open the file, load it into a buffer just to pass it into D3DXCreateTextureFromFileInMemoryEx is a loss of time, the function can do it for you. But the problem is not that. The problem is that you are reserving blocks of memory each time you do:
long fl = _filelength( fh );

But I don't see you releasing that memory. I'd recommend using a buffer with enough size to hold the data. That will be way faster.

But your worst problem is in the Filter and MipFilter parameters. The default filters can be really slow because they try to be a general solution in order to give you the best result when you try to stretch the texture on load time.

Say you are trying to load a texture with non pow2 dimensions, then DX will try to adjust it into a pow 2. I.E. If you load a 100x100 texture, DX will load it into a 128x128 texture. The vertical and horizontal relation with the original texture is set in the image_info structure. So, as the the bmp is being stretched, the filters are used in order to compensate. That makes your loading time slower.

In order to avoid the use of filters, create or pack your artwork using pow2 dimensions. Anyway, if your artist has created a texture, you can assume its a final and optimal version so any form of stretching will destroy the original presentation. So its wise to create your textures in pow2 formats. Once you do that the texture can be loaded using 0,0 in the Width and Height parameters and you will be sure they will have the same dimensions as your original art.

Now theres other big problem in your program. If you use 0 in the miplevels, a complete mipchain will be created for each texture. That means a 33% more in memory (no big deal), but the most important pat is that EACH miplevel is loaded stretching the image with the MipFilter you selected. If you read the rest of this message that means a lot of work if you aren't using mips.

Finally, using a color key is really slow. Try using a format that supports alpha like TGA, PNG or one of the DXTn formats. Colorkey usage is deprecated and you have a lot of better effects that can be achieved by using an alpha map. Usage of the color key slows you down because each image (and mipmap) should be processed in order to transform the color key into an alpha level.

So, I post you a suggested way to do this (in case you dont need mips and your textures are properly packaged):

D3DXIMAGE_INFO imageInfo;
HRESULT hr = E_FAIL;

// read from memory
hr = D3DXCreateTextureFromFileEx(m_pd3dDevice, buf,0,0,1,0,
D3DFMT_UNKNOWN, D3DPOOL_MANAGED, D3DX_FILTER_NONE,
D3DX_FILTER_NONE, 0, &imageInfo, NULL, &m_pTexture);
// save image dimensions
m_imageWidth=imageInfo.Width;
m_imageHeight=imageInfo.Height;


Luck!
Guimo

Share this post


Link to post
Share on other sites
streamer    415
Hi Guimo!

Thanx a lot! It speeds program very much! Seems to be a problem with mip-maps...
Anyway I noticed some weeks ago a 'bug' when I load textures which are not pow^2. It'll be changed soon.
Strangely my hd->memory and memory->texture type routine is still a little bit faster than hd based dx routine (and dxload routine made a mess on screen when snow started to fall in game (snow texture 32x32)
Memory is fred off course :) I did't paste it in my post.
Anyway first loading is a little bit longer, and then every other start is about 3 sec. Not bad :)
And one more question: is there any performance drawbacks if I use textures in 256 colors instead of 32bit textures?

Share this post


Link to post
Share on other sites
Guimo    463
Hi Streamer,

YEs, there is a penalty. Current HW has dropped support for color textures. You still can use it for alpha or other small tricks but not for color. That means colors must be transformed into your final representation in memory. be it a 16 bit A1R5G5B5 or 32 bit A8R8G8B8 (or others). Thats extra loading time.

I'd suggest using TGA files while testing where alpha is encoded in the alpha channel. After that wou can process your alpha and transform them into one of the DXTn formats when releasing.

Luck!
Guimo

Share this post


Link to post
Share on other sites
Nik02    4348
I would recommend DXT-compressed textures, packed into one big resource file. DXT doesn't need to be uncompressed, since the graphics hardware can use the compressed form directly, often without any performance penalty.

DXT techniques are lossy though - if you want absolutely best image quality, then it is good to use PNG:s (or HDR:s) anyway.

Share this post


Link to post
Share on other sites
MasterWorks    496
I used to have individual texture files for animation (say, 64 frames, 128x128 pixels) and after I consolidated each animation into a large (PNG) file, the game loads SO MUCH faster... the overhead for processing a file/texture is significant...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this