Sign in to follow this  

Large amount of Texture Data

This topic is 3332 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi, in my game using C++/DirectX9 I am loading multiple textures for objects, characters, etc. and it is beginning to slow down the game's load time. Right now the object/character sheets are 2048/1024 pixels and I load them all at the beginning of the game. It currently is like 5 seconds, but I only have a very small portion of the total planned art in the game. My question is, should I focus on trying to shrink and maximize the textures (shrinking them down to a smaller size, like 1024x1024 or 1024x512) or should I devise a better way of loading textures only when I need them? One idea I thought was to load a list of textures at run time, and when I need them, load only the file needed and then use it as needed. Loading textures as I need them, will that be too slow? Also do I just keep it in memory or release it after I have not needed it for awhile?

Share this post


Link to post
Share on other sites
Five seconds is fantastic! Loading a Crysis level on my PC takes a lot longer than that.

The simplest method is to use smaller textures during development (e.g. in a separate folder, so you can switch instantly). Make 128x128 versions of those 2048x2048 textures, and they will load instantly.

Share this post


Link to post
Share on other sites
How about coding a texture compiler tool?
Its really simple , here's how it will work:

Compiling :

-Load your texture as usual
-Create a binary file( f )
-Write header ( f)
{
-Write texture width ( f)
-Write texture height ( f )
-Write texture channels (f)
-Write texture buffer size
-Finally , write the pixel buffer
}



Loading a texture:


TextureCompiler_Header h;
loadCompiledTexture(&h);

h.m_Buffer contains image pixel data

Now , just assign the buffer to a texture ID , and you're ready to go.

For example , in opengl , you would have to do this:
gen textures
bind texture
set pixel unpack alignment
set parameters

glTexImage2D (GL_TEXTURE_2D, 0, h.m_Channels,h.m_Width, h.m_Height, 0, h.m_Channels==4 ? rgba : rgb , GL_UNSIGNED_BYTE,
h.m_Buffer
);




I'm sure this will give a big performance boost since
your loader won't have to loop through each pixel and generating the final
surface...it will just copy a pre-compiled image buffer.

Hope it helps...

Share this post


Link to post
Share on other sites
Quote:
Original post by Mattijs2
I would not call this 'compiling'. I'd call it 'creating your own texture image file format'. Does this solve anything?
Sometimes yes, sometimes no. Sometimes (particularly on laptops, which tend to have slow hard drives), hard drive access times will be your bottleneck, in which case you can speed things up by storing highly compressed images (such as JPEG). Other times image processing/decompression will be the bottleneck, so you store the images uncompressed on the HD, in the exact format to load up to the GPU, thus minimising processing.

Share this post


Link to post
Share on other sites
JPEG texture decompressing for textures takes a lot of time, at least at several machines I have tried. My personal opinion that best results are gained with PNG texture format, and you have compression and alpha channel. Also a slightly better performance is gained if you load file as binary format in memory then load texture directly from memory and not from HD.

Share this post


Link to post
Share on other sites
I remember that GTA 3 would convert all textures to an optimal format for my video card when I started it for the first time after installation. It took a lot of time... I have not seen any other game do this, however. not even Vice City.

Share this post


Link to post
Share on other sites
I am using PNG and thank you all for your tips.

I am not sure on how to approach this problem though - The solution 3d Gone Wild seems that it will take just as long, but I could be wrong.

Also, 5 seconds is now with a very small portion of the art. It will be much longer than that later in development.

Share this post


Link to post
Share on other sites
Quote:

The solution 3d Gone Wild seems that it will take just as long


No , you're wrong. Only the tool will need some tile to unpack/pack pixels...

Custom texture format vs PNG :

Loading a PNG:


open f
read "magic number"
check signature
create png read struct
create png info struct
init png io
set sig bytes
read nfo
get bit depth
get color tpe
handle color type
check if ptr is valid
get bit delpth
set strip
set packng
update info
get ihdr
allocate memory for texture
allocate memory for row pointers
loop through pixels
fill image
destroy struct
destroy row pointers
close file
return pixels




Custom texture file format:


open f
check header
(optional) get width
(optional) get height
(optional) get channels
read size of pixel buffer
allocate memory
f read offset , offset + pixel buffer size


Share this post


Link to post
Share on other sites
My favourite texture format is DDS. Compared to PNG and JPEG, it can:

- store mipmaps (calculating them on the fly does take a non-trivial amount of time)
- store DXT-compressed textures
- contain cubemaps
- support A2R10G10B10, ARGBF32, ARGBF16. R16 etc. pixel formats

Also, there are tons of tools for it available already, and writing a loader is not hard. The benefits mentioned above also mean that I can upload the texture data directly to memory, without decompressing or converting anything.

Share this post


Link to post
Share on other sites
Quote:
Original post by Crazyfool
I am using PNG and thank you all for your tips.

I am not sure on how to approach this problem though - The solution 3d Gone Wild seems that it will take just as long, but I could be wrong.

Also, 5 seconds is now with a very small portion of the art. It will be much longer than that later in development.


How are you loading your textures? Are they all loaded into a texture manager? Are they loaded in as each game object is loaded?

If you tie the texture loading to object loading you won't load unnecessary textures. You'll always have a big load at the start of a level/area, but if you're expecting it to shoot up into giant numbers you might be doing something wrong.

Share this post


Link to post
Share on other sites
Quote:
Original post by dv
My favourite texture format is DDS. Compared to PNG and JPEG, it can:

- store mipmaps (calculating them on the fly does take a non-trivial amount of time)
- store DXT-compressed textures
- contain cubemaps
- support A2R10G10B10, ARGBF32, ARGBF16. R16 etc. pixel formats

Also, there are tons of tools for it available already, and writing a loader is not hard. The benefits mentioned above also mean that I can upload the texture data directly to memory, without decompressing or converting anything.
Seconded. I suspect that using .dds files would be about as performant as using your own custom format, and means you don't need to spend time writing a file packer and unpacker.

Share this post


Link to post
Share on other sites
Alright thanks a lot for the helpful tips.

I'll look how to load DDS files (currently I just use the D3DXCreateTextureFromFileEx () for png images). I appreciate all your help!

Edit: I misread this. Looks like I can load these directly into the game how I have been, but I should create a loader to smooth the transition (is this correct?)

Share this post


Link to post
Share on other sites
I can load dds files with CreateTextureFromFileEx() right? But that alone wont solve my problems, right? I misunderstood that.

I am looking into figuring out how to do a texture loader (have been dealing with other bugs, so have not started on this yet, but I know the general direction which is why I made this post).

Thanks again!

Share this post


Link to post
Share on other sites
Quote:
Original post by Crazyfool
Hi, in my game using C++/DirectX9 I am loading multiple textures for objects, characters, etc. and it is beginning to slow down the game's load time. Right now the object/character sheets are 2048/1024 pixels and I load them all at the beginning of the game. It currently is like 5 seconds, but I only have a very small portion of the total planned art in the game.


Which says nothing of actual problems.

How many images? How many files? How many megabytes total? What kind of CPU? How much memory?

If you have large number of files (thousands) then especially under Windows, packing all those into single file (tar or uncompressed zip work just fine) will typically bring considerable reduction in load times.

If you have large amount of data that needs to be read (hundreds of megabytes) then compressing individual images may help. Choice of format here depends on many factors. Assume 5-50Mb/sec for disk reads.

If your assets are designed to work on specific and well known set of hardware, storing data in a way that matches memory layout may bring minor performance improvements. For anything semi-portable that isn't a preferred option, or it's hardly worth the hassle.

If you're using some slow media (CD/DVD), laying out the files in a certain order may help.

But above all, determine how much work you need to do first.


But, eventually you will hit a wall. There, the usual choice is to use asynchronous loading, and load optional assets while you're already running the scene.

Share this post


Link to post
Share on other sites
Thanks a lot Antheus for your input.

Basically I have about 20 (if I add 1 or 2, it doesnt change it noticeably at all, at least not to me, always around 4-6 seconds) images that are 1024 x 2048 PNG images. I also have like 60 PNG images that are much smaller, all separate files. Load time while only loading 1-3 of the 'bigger' images is cut down to a second or less.

Here is how I am doing it:

D3DXCreateTextureFromFileEx(d3ddev,
fname.c_str(),
D3DX_DEFAULT,
D3DX_DEFAULT,
D3DX_DEFAULT,
NULL,
D3DFMT_A8R8G8B8,
D3DPOOL_MANAGED,
D3DX_DEFAULT,
D3DX_DEFAULT,
D3DCOLOR_XRGB(255, 0, 255),
NULL,
NULL,
&texture);

Share this post


Link to post
Share on other sites
Quote:
Original post by Crazyfool
I can load dds files with CreateTextureFromFileEx() right? But that alone wont solve my problems, right? I misunderstood that.

I am looking into figuring out how to do a texture loader (have been dealing with other bugs, so have not started on this yet, but I know the general direction which is why I made this post).

Thanks again!


I have no idea about DirectX , but actually it doesn't matter.
Its really easy to do it , it works well , and you'll get better loading times.

Here's how i've done it in the past (pseudo code)

Packing:

ImageStructure* img = functionToLoadPngImageFromFile;
open binary file out
write header structure
{
img->w
img->h
img->channels
buffer size ( img->w * img->h * img->channels )
}
write buffer ( img->data , header.buffersize )

we're done...






Loading:


TPackedTextureHeader t;
open binary file f
read ( (char*)(&t) , sizeof(TPackedTextureHeader) );
unsigned char* pixels = new unsigned char [ t.buffer size ];
//now just read the buffer without even moving in file
read(pixels,t.buffer size);
bind pixels to a texture
delete [] pixels






Of course things can get a little bit more complicated if you're going to create
a "real packer" were you can pack more than 1 textures into 1 single file.

Share this post


Link to post
Share on other sites

This topic is 3332 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this