What modern, open-source compression library should I use

Started by
31 comments, last by frob 6 years, 9 months ago
23 minutes ago, Hodgman said:

If you want to avoid conversion, pre-convert your textures into GPU-compatible data formats (i.e. anything in this list) ahead of time. Note that RGB and BGR are not present in that list, but RGBA and BGRA are present (DXGI_FORMAT_R8G8B8A8_UNORM_SRGB / DXGI_FORMAT_B8G8R8A8_UNORM_SRGB). Also note that the DXGI_FORMAT_BC* (AKA DXT) formats are compressed and remain compressed on the GPU! Ideally you would pre-convert all your textures into BC/DXT format and save them as .dds files (and then perhaps put the dds files into a compressed archive).

I was using TGA, but I'll go ahead and try to convert to DDS. I can't do it right now, as I'm rewriting my graphics module to be more flexible and gradually degrade in terms of drawing techniques, but I'll work on all this afterwards.

25 minutes ago, Hodgman said:

I'm guessing you're using stbi_loadf? An easy win without rearranging too much stuff will be to use stbi_loadf_from_memory after already streaming the data into memory yourself. Don't use the C file loading API as it's synchronous/blocking, and ties up a whole thread while it waits on disk. Use your OS's native async file loading API to stream files in the background, poll once per frame to see if they're complete, and when they are, call stbi_loadf_from_memory. This advice holds true regardless of whether you're using loose-files, file-archives, compression, or no compression.

I hadn't considering async for file loading but I'll look into it.

26 minutes ago, Hodgman said:

Also pre-generate a list of textures that a model requires, so that you can begin loading ALL of those textures (in the background) right away as soon as the model is loaded.

The material and shader format I'm designing will be loaded along with files with the same name, and then report the shader files for the graphics library I'm going after, as well as name the uniform data I can pass to it including textures, color data, etc, which will be handled by the material files so it's more flexible. The material files will contain a list of all textures, I'll load a list of all the textures I need for the entire scene and then load them all, so that should fit the list you were referring to.

Advertisement

I hope you realize there are reasons behind these formats. 

5 hours ago, KarimIO said:

I was using TGA, but I'll go ahead and try to convert to DDS.

TGA is a very old image format that is lossless and supports an unlimited block of custom data.  The big strength (extremely valuable) is the ability to store arbitrary metadata, such as other image formats or source data. The biggest drawback is the size.

DDS is a family of compressed lossy data formats. The big strength is video cards can load the data directly to video cards without processing.  The biggest drawback is the lossy nature of compression, they should never be edited, only rebuilt from source assets.

Games will often have multiple versions of art and texture assets.  They may have a photoshop document (PSD), some intermediate files like PNG or TGA, and a final output format of DDS or PVRTC or whatever can be directly used by the hardware.

Be careful to use the right image format for the job.  Web sites love jpeg and png files because they can be tightly compressed, but for games that means time and memory spent decompressing them. 

This topic is closed to new replies.

Advertisement