Quote:Original post by phantom
To be fair your requirements differ from those of games;

I don't think they do, actually. Although the style of the game (and its textures) surely matter, I think that it mostly depends on the player. While the intensity of texture artifacts generated by DXTC is definitely dependent on the type of textures you use, there is a lot of visual psychology involved as well.

Now, let's not kid ourselves here: DXTC is a horrible compression. The first and foremost priority with the development of DXTC was easy and efficient hardware implementation on GPUs that are now many generations old. It was not quality. Compared to other lossy systems, such as DCT or DWT based ones, DXTC is unbelievably primitive. But it can be decoded easily by GPUs, while DCT/DWT cannot (yet). This is a huge advantage - albeit its only one.

DXTC cannot be used to store work-in-progress assets during development. You need a lossless format for that, otherwise you'll kill your texture in no time due to entropy. Sure you could use DDS in lossless mode, but PNG is better in that regard (higher compression ratio, wider adoption, etc). So, if you want your engine to load compressed DDS files, you have to run a 'compilation' step on all your textures. This adds an additional time consuming and potentially error prone step to the build pipeline. If you use a lossless format, you can directly load your work textures during development, almost directly out of Photoshop. Much more convenient.

Of course, the question remains if textures should be compressed in VRAM or not. I think that this choice should be left to the player. People react very differently on compression artifacts. Some will never see them, others will be driven crazy by them. By storing textures as compressed DDS on disk, you force everyone to use the textures that you deemed compression-worthy. Even if you don't see a problem with DXTC'ing a particular image, someone else might be annoyed by the artifacts. The human brain is a strange machine :)

Also keep in mind that some shaders will greatly enhance the visibility of artifacts. Even diffuse maps that look good with DXTC in a raw state can suddenly look terrible with a particular shader applied.

So I make it a preference switch. You don't lose anything, and gain a lot. If your players upgrade their graphics card, they can turn off compression and your game will look better without any additional costs. Users with lower end cards just turn DXTC on. If some day someone manages to create a better VRAM compression algorithm either in hardware or through shaders, using it in your game is the matter of changing a few lines of code. No need to rebuild any asset.

Let me dispel another myth, that offline DXTC compression looks better than on-the-fly. This is not true. It used to be true when people compared offline with the drivers' built-in compression, which could indeed have a very bad quality. Nowadays, the driver compression has improved a lot. And if you don't like it, just use your own implementation while loading. The overhead is minimal when using multi-core plus SIMD, and the quality is practically identical to offline compressors for most of the (many) tests we did.

So in the end, I see DXTC as a necessary evil to conserve VRAM and texture access bandwidth. However, I don't see it as an appropriate format to store your textures in, because much better algorithms exist for that. Use the appropriate tool for any given job.

Quote:Original post by phantom

Quote:Original post by Yann L
Besides that, I'm not sure how DXTC would actually compress. DXTC is a pretty crude compression system design for speed rather than quality. It generates a lot of quantization errors that could limit the way a lossless compressor may process the data. But I never tried it. Could be interesting to do.

So, yeah, YMMV but it looks like zip compression with DXTn compressed files is a good bet.

The results are interesting. So, if you're very tight on storage space and/or have specific restrictions in that regard, then using DXTC+zip is indeed a good bet.

However, to be entirely fair, you'd have to consider JPEG as well as a storage format. Or even a DWT-based lossy compression. The figures would look entirely different then.

Quote:Original post by Yann L
DXTC cannot be used to store work-in-progress assets during development. You need a lossless format for that, otherwise you'll kill your texture in no time due to entropy.

I would argue that's what PSDs are for.

Quote:
So, if you want your engine to load compressed DDS files, you have to run a 'compilation' step on all your textures.

You should be doing this anyway. As has been pointed out before in this thread, 100 file accesses from disk is very bad for load times. At the very least you want to dump all your files in a single archive (even a ZIP will work).

Quote:
Of course, the question remains if textures should be compressed in VRAM or not. I think that this choice should be left to the player. People react very differently on compression artifacts. Some will never see them, others will be driven crazy by them. By storing textures as compressed DDS on disk, you force everyone to use the textures that you deemed compression-worthy. Even if you don't see a problem with DXTC'ing a particular image, someone else might be annoyed by the artifacts. The human brain is a strange machine :)

I would argue a different way: DXT let's you use higher resolution textures with the same file size as a smaller resolution model. That is, if I use DXT5, I can use a 1024x1024 image where a 512x512 image was before, without any additional costs in terms of performance or VRAM. DXT1 even more so. Do the compression artifacts outweigh the increase in quality from quadrupling the size of your texture? (Not rhetorical, I'm curious actually)

You could of course store the 1024x1024 textures in the game assets directory, and compress them or not at load time depending on user preferences, but it just seems like a lot of extra work to be doing during level load. Maybe if you made it an installation option. But then, you remove the ability for your artists to customize which DXT version to use. Sometimes you can get away with DXT1, sometimes you need a full 32 bit mode.

Probably the optimal version would let the artists specify an optional compression format in an options file for every texture. Then store two versions of every texture (DXT compressed or not), and let users decide which/both versions to install. But it seems like a lot of extra work still.

I think you want to avoid doing something like compressing every texture as DXT5. The different DXT versions have different strengths and weaknesses, and should really be decided by a human at some point.

Quote:Original post by phantom
The results are interesting. So, if you're very tight on storage space and/or have specific restrictions in that regard, then using DXTC+zip is indeed a good bet.

Remember what you said your bottleneck was: hard drive access speed. That's true for every game I've ever seen (well, except for that FPS that used procedural content for everything and had an install under a couple KBs). You zip the DDSs for loading speed, nothing else really. If you can clip off 20% on average from your DDS textures, that might mean a 20% reduction in load times. Which are probably too long as it is anyway.

Quote:

Quote:
So, if you want your engine to load compressed DDS files, you have to run a 'compilation' step on all your textures.

You should be doing this anyway.

For the shipping product, of course. But not during development. We used to do that, but it created a lot of annoyances and was too inflexible. The new approach has significantly increased productivity due to better rapid prototyping capabilities.

Quote:
I would argue a different way: DXT let's you use higher resolution textures with the same file size as a smaller resolution model. That is, if I use DXT5, I can use a 1024x1024 image where a 512x512 image was before, without any additional costs in terms of performance or VRAM.

Not entirely correct. Texture fetch latency and caching behaviour will be (slightly) affected, and due to more mipmap levels you'll inevitably increase memory usage (assuming DXT5).

Quote:
DXT1 even more so. Do the compression artifacts outweigh the increase in quality from quadrupling the size of your texture? (Not rhetorical, I'm curious actually)

Usually the increased resolution makes up for the artifacts. But not always. As I said above, it depends on the contents of the texture and on the person viewing it.

However, this was not my point. When you design your game and all artwork assets, you define a common lowest denominator for it to work on. Say you target 512MB VRAM class hardware. You will choose your resolutions and compression settings so to avoid (too much) swapping on such a card. Now, imagine some user buying a new 1GB VRAM card. Essentially, since your game was designed with 512MB in mind, half of his memory will be unused and the textures will look the same as on a 512MB card. Mr.NewCardOwner will not be happy. Now, of course you could distribute your game with a high resolution texture pack, but this come at a cost since it increases the size on disk and means additional work for the artists, who now have to maintain two different versions.

The alternative is to let him check a box called 'uncompressed textures', and load all (or only critical) textures without compression. It could look much nicer than a highres pack. We don't know. We can't know. Because again, the perception of compression artifacts is *very* variable from one person to the next. And doing so doesn't really cost you anything, except for a PNG loader (readily available) and a (possibly multithreaded) DXTC compressor (not too difficult to write).

Quote:
You could of course store the 1024x1024 textures in the game assets directory, and compress them or not at load time depending on user preferences, but it just seems like a lot of extra work to be doing during level load.

The overhead is minimal, as I said. The quality improvement without DXTC can be significant depending on your textures. But you don't lose the capability of enabling DXTC if needed. The only difference is that this decision is now done by the player, and not by you. Let him choose the quality versus performance tradeoff.

Quote:
Probably the optimal version would let the artists specify an optional compression format in an options file for every texture.

I wouldn't trust an average artist to handle that [wink]

Quote:
Remember what you said your bottleneck was: hard drive access speed. That's true for every game I've ever seen (well, except for that FPS that used procedural content for everything and had an install under a couple KBs). You zip the DDSs for loading speed, nothing else really. If you can clip off 20% on average from your DDS textures, that might mean a 20% reduction in load times. Which are probably too long as it is anyway.

If you think like that, then why not consider JPEG instead ? It's going to be smaller, after all. Of course there's the JPEG->DXTC transcoding, which would suck. But seriously, while the texture loading is HDD limited, it only makes a small percentage of the overall loading time of a scene.

I just can't understand how people can store their assets in a format that was never meant to be a lossy image storage format. It was designed as a compression with a GPU-friendly decoding process. It is good at doing this for now, but why use it for something where other, better alternatives exist ?

As a counter example, consider JPEG. That's a lossy format that was specifically designed for image storage. It cannot be decoded on a GPU (not yet). But it is both significantly smaller and higher quality than DXTC for stored images. Because that's what it was designed for.

Oh well, I'm actually a little obsessed with quality. I'm one of these people who find 128kbs MP3s absolutely unbearable, who are incredibly annoyed when the friends you were on holidays with send you some memorable snapshots as jpegs with a 60% quality level. That's why I cringe at the mere thought of someone using such a terrible compression system as DXTC to store textures. We all want full 32bit pixel shaders nowadays, we store normalmaps with 16bit per component precision, and so on. Yet we brutalize our textures in such a way without an option to switch it off if you don't need it ? Nope, not for me [smile]

Quote:Original post by Kest

Quote:Also, while PNG might well be compressed it is only compressed on disk; DXT compressed images remain compressed in texture ram

Any format can be compressed once its in system ram.
IIRC, DirectX actually has the ability to load a PNG and save it as DDS, or vice versa. Once the texture is in ram, all format bets are off.

^^fixed to show apples and oranges ;)

If you're loading in one format and performing a conversion in system ram, then it increases your load texture processing times, which may increase load times. DXT is good because you can pass data straight from disc through to the graphics card with no conversions (assuming you haven't done something clever like ZIP up your textures ;) )

[Edited by - Hodgman on August 28, 2008 11:39:48 PM]

Quote:Original post by Zipster
Like all things, it's a trade-off, so it really isn't fair to make statements like "Why can't everyone just do things like X?" We want to but we can't! [wink]

Well, you brought up a valid reason to use DXTC precompressed textures. You are absolutely right, sometimes you don't have the choice, and sometimes it actually is the right thing to do. On the other hand, many people, especially indie developers, have much more freedom of choosing potentially superior alternatives. That's why I was a little surprised that some people here presented DXTC precompressed assets as the only and optimal way. I found it also very misleading that people made it look like using compressed DDS files was the only way of getting DXTC compression on the GPU - which is blatantly wrong.

Quote:Original post by Hodgman
If you're loading in one format and performing a conversion in system ram, then it increases your load times.

No, it doesn't have to. In fact, it can actually decrease loading times.

Quote:Original post by Yann LNo, it doesn't have to. In fact, it can actually decrease loading times.

You're right; I should qualify that by saying it increases processing time, which may not be a problem if the disc is the bottleneck ;)