Jump to content

  • Log In with Google      Sign In   
  • Create Account


Why do game companies use custom texture formats instead of the DDS format?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
85 replies to this topic

#41 phantom   Moderators   -  Reputation: 6871

Like
0Likes
Like

Posted 28 August 2008 - 11:14 AM

Yann;

To be fair your requirements differ from those of games; indeed I even said that not all textures would need DXT compression.

Offline compression could also benfit from improved quality and of course more control, certainly when it comes to compressing the smaller mipmaps of the image.

But with all things no 'one size' fits all.


Quote:
Original post by Yann L
Besides that, I'm not sure how DXTC would actually compress. DXTC is a pretty crude compression system design for speed rather than quality. It generates a lot of quantization errors that could limit the way a lossless compressor may process the data. But I never tried it. Could be interesting to do.


I guess this would depend on the image as much as anything. For example given this image;

(256*256)
A 32bit PNG comes in at 7,734bytes (8,192 on disk)
DXT1 32,896bytes (36,864)
DXT3 65,664bytes (69,632)
DXT5 65,664bytes (69,632)
Now, if the DXTs are compressed via WinRAR in zip format using "Best" compression (with one file per zip) the resulting sizes (taken from WinRAR) are;
PNG 7,561 bytes
DXT1 2,738 bytes
DXT3 3,296 bytes
DXT5 3,495 bytes
With added mipmaps (9 levels);
DXT1 43,832 bytes (45,056)
DXT3 87,536 bytes (90,112)
DXT5 87,536 bytes (90,112)
And compressed as before;
DXT1 5,000 bytes
DXT3 6,029 bytes
DXT5 6,634 bytes

That image however it pretty compression friendly, with a slightly more unfriendly image such as;

(again, 256*256)
The results are;
A 32bit PNG comes in at 142,580 bytes (143,360 on disk)
DXT5 65,664bytes (69,632)
DXT5 with mipmaps 87,536bytes (90,112)
Zip compressed;
PNG 142,605 (!bigger!)
DXT5 33,287 bytes
DXT5 with mipmaps 44,227 bytes

So, yeah, YMMV but it looks like zip compression with DXTn compressed files is a good bet.

Sponsor:

#42 dietepiet   Members   -  Reputation: 157

Like
0Likes
Like

Posted 28 August 2008 - 11:43 AM

I know that the commercial game 'Faces of War' uses DDS textures for there model textures. And I know this simply by looking ;)
From experience I can say: Yes, if you want to, you can reverse the data formats of most games. But until now, I only ripped data for my own projects from games that where weakly protected, because there are enough weak protected games around to not wast your time reverse engineering encrypted file formats.

But of course if you ask me, I'll always say that you shouldn't take the effort to protect your data because I personally am very curious :D

#43 Yann L   Moderators   -  Reputation: 1794

Like
0Likes
Like

Posted 28 August 2008 - 11:58 AM

Quote:
Original post by phantom
To be fair your requirements differ from those of games;

I don't think they do, actually. Although the style of the game (and its textures) surely matter, I think that it mostly depends on the player. While the intensity of texture artifacts generated by DXTC is definitely dependent on the type of textures you use, there is a lot of visual psychology involved as well.

Now, let's not kid ourselves here: DXTC is a horrible compression. The first and foremost priority with the development of DXTC was easy and efficient hardware implementation on GPUs that are now many generations old. It was not quality. Compared to other lossy systems, such as DCT or DWT based ones, DXTC is unbelievably primitive. But it can be decoded easily by GPUs, while DCT/DWT cannot (yet). This is a huge advantage - albeit its only one.

DXTC cannot be used to store work-in-progress assets during development. You need a lossless format for that, otherwise you'll kill your texture in no time due to entropy. Sure you could use DDS in lossless mode, but PNG is better in that regard (higher compression ratio, wider adoption, etc). So, if you want your engine to load compressed DDS files, you have to run a 'compilation' step on all your textures. This adds an additional time consuming and potentially error prone step to the build pipeline. If you use a lossless format, you can directly load your work textures during development, almost directly out of Photoshop. Much more convenient.

Of course, the question remains if textures should be compressed in VRAM or not. I think that this choice should be left to the player. People react very differently on compression artifacts. Some will never see them, others will be driven crazy by them. By storing textures as compressed DDS on disk, you force everyone to use the textures that you deemed compression-worthy. Even if you don't see a problem with DXTC'ing a particular image, someone else might be annoyed by the artifacts. The human brain is a strange machine :)

Also keep in mind that some shaders will greatly enhance the visibility of artifacts. Even diffuse maps that look good with DXTC in a raw state can suddenly look terrible with a particular shader applied.

So I make it a preference switch. You don't lose anything, and gain a lot. If your players upgrade their graphics card, they can turn off compression and your game will look better without any additional costs. Users with lower end cards just turn DXTC on. If some day someone manages to create a better VRAM compression algorithm either in hardware or through shaders, using it in your game is the matter of changing a few lines of code. No need to rebuild any asset.

Let me dispel another myth, that offline DXTC compression looks better than on-the-fly. This is not true. It used to be true when people compared offline with the drivers' built-in compression, which could indeed have a very bad quality. Nowadays, the driver compression has improved a lot. And if you don't like it, just use your own implementation while loading. The overhead is minimal when using multi-core plus SIMD, and the quality is practically identical to offline compressors for most of the (many) tests we did.

So in the end, I see DXTC as a necessary evil to conserve VRAM and texture access bandwidth. However, I don't see it as an appropriate format to store your textures in, because much better algorithms exist for that. Use the appropriate tool for any given job.

Quote:
Original post by phantom
Quote:
Original post by Yann L
Besides that, I'm not sure how DXTC would actually compress. DXTC is a pretty crude compression system design for speed rather than quality. It generates a lot of quantization errors that could limit the way a lossless compressor may process the data. But I never tried it. Could be interesting to do.

So, yeah, YMMV but it looks like zip compression with DXTn compressed files is a good bet.

The results are interesting. So, if you're very tight on storage space and/or have specific restrictions in that regard, then using DXTC+zip is indeed a good bet.

However, to be entirely fair, you'd have to consider JPEG as well as a storage format. Or even a DWT-based lossy compression. The figures would look entirely different then.

#44 Numsgil   Members   -  Reputation: 501

Like
0Likes
Like

Posted 28 August 2008 - 12:28 PM

Quote:
Original post by Yann L
DXTC cannot be used to store work-in-progress assets during development. You need a lossless format for that, otherwise you'll kill your texture in no time due to entropy.


I would argue that's what PSDs are for.

Quote:

So, if you want your engine to load compressed DDS files, you have to run a 'compilation' step on all your textures.


You should be doing this anyway. As has been pointed out before in this thread, 100 file accesses from disk is very bad for load times. At the very least you want to dump all your files in a single archive (even a ZIP will work).

Quote:

Of course, the question remains if textures should be compressed in VRAM or not. I think that this choice should be left to the player. People react very differently on compression artifacts. Some will never see them, others will be driven crazy by them. By storing textures as compressed DDS on disk, you force everyone to use the textures that you deemed compression-worthy. Even if you don't see a problem with DXTC'ing a particular image, someone else might be annoyed by the artifacts. The human brain is a strange machine :)


I would argue a different way: DXT let's you use higher resolution textures with the same file size as a smaller resolution model. That is, if I use DXT5, I can use a 1024x1024 image where a 512x512 image was before, without any additional costs in terms of performance or VRAM. DXT1 even more so. Do the compression artifacts outweigh the increase in quality from quadrupling the size of your texture? (Not rhetorical, I'm curious actually)

You could of course store the 1024x1024 textures in the game assets directory, and compress them or not at load time depending on user preferences, but it just seems like a lot of extra work to be doing during level load. Maybe if you made it an installation option. But then, you remove the ability for your artists to customize which DXT version to use. Sometimes you can get away with DXT1, sometimes you need a full 32 bit mode.

Probably the optimal version would let the artists specify an optional compression format in an options file for every texture. Then store two versions of every texture (DXT compressed or not), and let users decide which/both versions to install. But it seems like a lot of extra work still.

I think you want to avoid doing something like compressing every texture as DXT5. The different DXT versions have different strengths and weaknesses, and should really be decided by a human at some point.

Quote:
Original post by phantom
The results are interesting. So, if you're very tight on storage space and/or have specific restrictions in that regard, then using DXTC+zip is indeed a good bet.


Remember what you said your bottleneck was: hard drive access speed. That's true for every game I've ever seen (well, except for that FPS that used procedural content for everything and had an install under a couple KBs). You zip the DDSs for loading speed, nothing else really. If you can clip off 20% on average from your DDS textures, that might mean a 20% reduction in load times. Which are probably too long as it is anyway.
Darwinbots - Artificial life simulation

#45 Yann L   Moderators   -  Reputation: 1794

Like
0Likes
Like

Posted 28 August 2008 - 01:36 PM

Quote:

Quote:

So, if you want your engine to load compressed DDS files, you have to run a 'compilation' step on all your textures.

You should be doing this anyway.

For the shipping product, of course. But not during development. We used to do that, but it created a lot of annoyances and was too inflexible. The new approach has significantly increased productivity due to better rapid prototyping capabilities.

Quote:

I would argue a different way: DXT let's you use higher resolution textures with the same file size as a smaller resolution model. That is, if I use DXT5, I can use a 1024x1024 image where a 512x512 image was before, without any additional costs in terms of performance or VRAM.

Not entirely correct. Texture fetch latency and caching behaviour will be (slightly) affected, and due to more mipmap levels you'll inevitably increase memory usage (assuming DXT5).

Quote:

DXT1 even more so. Do the compression artifacts outweigh the increase in quality from quadrupling the size of your texture? (Not rhetorical, I'm curious actually)

Usually the increased resolution makes up for the artifacts. But not always. As I said above, it depends on the contents of the texture and on the person viewing it.

However, this was not my point. When you design your game and all artwork assets, you define a common lowest denominator for it to work on. Say you target 512MB VRAM class hardware. You will choose your resolutions and compression settings so to avoid (too much) swapping on such a card. Now, imagine some user buying a new 1GB VRAM card. Essentially, since your game was designed with 512MB in mind, half of his memory will be unused and the textures will look the same as on a 512MB card. Mr.NewCardOwner will not be happy. Now, of course you could distribute your game with a high resolution texture pack, but this come at a cost since it increases the size on disk and means additional work for the artists, who now have to maintain two different versions.

The alternative is to let him check a box called 'uncompressed textures', and load all (or only critical) textures without compression. It could look much nicer than a highres pack. We don't know. We can't know. Because again, the perception of compression artifacts is *very* variable from one person to the next. And doing so doesn't really cost you anything, except for a PNG loader (readily available) and a (possibly multithreaded) DXTC compressor (not too difficult to write).

Quote:

You could of course store the 1024x1024 textures in the game assets directory, and compress them or not at load time depending on user preferences, but it just seems like a lot of extra work to be doing during level load.

The overhead is minimal, as I said. The quality improvement without DXTC can be significant depending on your textures. But you don't lose the capability of enabling DXTC if needed. The only difference is that this decision is now done by the player, and not by you. Let him choose the quality versus performance tradeoff.

Quote:

Probably the optimal version would let the artists specify an optional compression format in an options file for every texture.

I wouldn't trust an average artist to handle that [wink]

Quote:

Remember what you said your bottleneck was: hard drive access speed. That's true for every game I've ever seen (well, except for that FPS that used procedural content for everything and had an install under a couple KBs). You zip the DDSs for loading speed, nothing else really. If you can clip off 20% on average from your DDS textures, that might mean a 20% reduction in load times. Which are probably too long as it is anyway.

If you think like that, then why not consider JPEG instead ? It's going to be smaller, after all. Of course there's the JPEG->DXTC transcoding, which would suck. But seriously, while the texture loading is HDD limited, it only makes a small percentage of the overall loading time of a scene.

I just can't understand how people can store their assets in a format that was never meant to be a lossy image storage format. It was designed as a compression with a GPU-friendly decoding process. It is good at doing this for now, but why use it for something where other, better alternatives exist ?

As a counter example, consider JPEG. That's a lossy format that was specifically designed for image storage. It cannot be decoded on a GPU (not yet). But it is both significantly smaller and higher quality than DXTC for stored images. Because that's what it was designed for.

Oh well, I'm actually a little obsessed with quality. I'm one of these people who find 128kbs MP3s absolutely unbearable, who are incredibly annoyed when the friends you were on holidays with send you some memorable snapshots as jpegs with a 60% quality level. That's why I cringe at the mere thought of someone using such a terrible compression system as DXTC to store textures. We all want full 32bit pixel shaders nowadays, we store normalmaps with 16bit per component precision, and so on. Yet we brutalize our textures in such a way without an option to switch it off if you don't need it ? Nope, not for me [smile]


#46 Kest   Members   -  Reputation: 547

Like
0Likes
Like

Posted 28 August 2008 - 03:45 PM

Sorry, I wasn't able to jump in when I should have responded.

Quote:
Original post by frob
Quote:
Original post by Kest
On the protection front, has anyone considered that obscuring/hiding data may be beneficial to players? Letting them easily browse through your assets is essentially like allowing theater play-goers to see your actors dressing up for their parts.

Everyone is curious, but not everyone really wants to see.

Quite a few games do provide them in easy formats.

Many games, even from megacorps and major publishers, have audio files directly in mp3, wma, ogg, or other formats, ready for your listening pleasure. If you look on console game disks, you can sometimes find these.

Being exposed to the internal resource assets of your game will almost certainly make the objects that are built with those assets feel more fabricated to your players. Your characters will seem less alive, and your world will seem more nuts & bolts. A high number of games ignoring that issue doesn't make it any less damaging. Especially in this era, where every game player is an aspiring game developer.

Quote:
Original post by phantom
What applications don't support DDS then?

Paint Shop Pro was one. Not sure, but I think 3D studio as well. I found a plugin for PSP, but it was pretty cumbersome. Something about the way it handled mip map viewing/editing.

However, I don't really want to add to the debate. At the time I chose PNG, DDS wasn't even around. PNG was the only 24+ bit transparency-capable format I could find that saved and loaded identically in every application. TGA and others seemed to be handled differently, depending on the application.

Quote:
Also, while PNG might well be compressed it is only compressed on disk; DXT compressed images remain compressed in texture ram

Any format can be compressed once its in ram. IIRC, DirectX actually has the ability to load a PNG and save it as DDS, or vice versa. Once the texture is in ram, all format bets are off.

#47 Zipster   Crossbones+   -  Reputation: 580

Like
0Likes
Like

Posted 28 August 2008 - 04:30 PM

Quote:
Original post by Yann L
I just can't understand how people can store their assets in a format that was never meant to be a lossy image storage format. It was designed as a compression with a GPU-friendly decoding process. It is good at doing this for now, but why use it for something where other, better alternatives exist ?

In an ideal world, you'd store and ship all your textures uncompressed and have the user chose whether or not they want to use a compressed version if they have a lower end graphics card. But at the end of the day, your shipped title has to fit on X (which is often 1) many DVDs. The space difference between uncompressed and compressed textures is actually quite significant. On our last game it was somewhere around a few gigabytes, and with all the other large assets on disk (bink movies, sounds, music, models, etc.) something had to give. At that point it makes a lot of sense just to go with something like DXT5, since it's a native DirectX format that can easily be loaded and passed directly to the hardware. No need to load and decompress another format, then potentially re-encode it to DXTC, which increases load times (that we didn't have) either way. Like all things, it's a trade-off, so it really isn't fair to make statements like "Why can't everyone just do things like X?" We want to but we can't! [wink]

#48 Hodgman   Moderators   -  Reputation: 28500

Like
0Likes
Like

Posted 28 August 2008 - 04:39 PM

Quote:
Original post by Kest
Quote:
Also, while PNG might well be compressed it is only compressed on disk; DXT compressed images remain compressed in texture ram

Any format can be compressed once its in system ram.
IIRC, DirectX actually has the ability to load a PNG and save it as DDS, or vice versa. Once the texture is in ram, all format bets are off.
^^fixed to show apples and oranges ;)

If you're loading in one format and performing a conversion in system ram, then it increases your load texture processing times, which may increase load times. DXT is good because you can pass data straight from disc through to the graphics card with no conversions (assuming you haven't done something clever like ZIP up your textures ;) )

[Edited by - Hodgman on August 28, 2008 11:39:48 PM]

#49 Yann L   Moderators   -  Reputation: 1794

Like
0Likes
Like

Posted 28 August 2008 - 04:52 PM

Quote:
Original post by Zipster
Like all things, it's a trade-off, so it really isn't fair to make statements like "Why can't everyone just do things like X?" We want to but we can't! [wink]

Well, you brought up a valid reason to use DXTC precompressed textures. You are absolutely right, sometimes you don't have the choice, and sometimes it actually is the right thing to do. On the other hand, many people, especially indie developers, have much more freedom of choosing potentially superior alternatives. That's why I was a little surprised that some people here presented DXTC precompressed assets as the only and optimal way. I found it also very misleading that people made it look like using compressed DDS files was the only way of getting DXTC compression on the GPU - which is blatantly wrong.

Quote:
Original post by Hodgman
If you're loading in one format and performing a conversion in system ram, then it increases your load times.

No, it doesn't have to. In fact, it can actually decrease loading times.

#50 Hodgman   Moderators   -  Reputation: 28500

Like
0Likes
Like

Posted 28 August 2008 - 05:01 PM

Quote:
Original post by Yann LNo, it doesn't have to. In fact, it can actually decrease loading times.
You're right; I should qualify that by saying it increases processing time, which may not be a problem if the disc is the bottleneck ;)

#51 Numsgil   Members   -  Reputation: 501

Like
0Likes
Like

Posted 28 August 2008 - 07:40 PM

Quote:
Original post by Yann L
Quote:

I would argue a different way: DXT let's you use higher resolution textures with the same file size as a smaller resolution model. That is, if I use DXT5, I can use a 1024x1024 image where a 512x512 image was before, without any additional costs in terms of performance or VRAM.

Not entirely correct. Texture fetch latency and caching behaviour will be (slightly) affected, and due to more mipmap levels you'll inevitably increase memory usage (assuming DXT5).


How so? If the compressed, but larger resolution texture is the same size in memory as an uncompressed, smaller version. Also, I don't think it'll increase memory usage even counting the mip map levels (compared with an uncompressed but smaller texture + its mip maps). Or if it does, it's like 4 bytes for the one extra level :)

Quote:

However, this was not my point. When you design your game and all artwork assets, you define a common lowest denominator for it to work on. Say you target 512MB VRAM class hardware. You will choose your resolutions and compression settings so to avoid (too much) swapping on such a card. Now, imagine some user buying a new 1GB VRAM card. Essentially, since your game was designed with 512MB in mind, half of his memory will be unused and the textures will look the same as on a 512MB card. Mr.NewCardOwner will not be happy.


In your scenario you're doing exactly what I'm saying. You're essentially designing your game with imaginary future tech cards in mind, with large texture sizes, and allowing current lowest denominator cards to run by compressing the textures.

Quote:

Quote:

Probably the optimal version would let the artists specify an optional compression format in an options file for every texture.

I wouldn't trust an average artist to handle that [wink]


Really? Is it that you're afraid the artists won't ever choose the compression because they don't want their art to have artifacts?

Quote:

If you think like that, then why not consider JPEG instead ? It's going to be smaller, after all. Of course there's the JPEG->DXTC transcoding, which would suck. But seriously, while the texture loading is HDD limited, it only makes a small percentage of the overall loading time of a scene.


What else are you doing in your scene? The time it takes to dump stuff (textures, models, etc.) from disk to GPU should be the bottleneck. If other stuff is slowing your loading times down, I can't help but feel you're doing something wrong. I can't imagine anything CPU intensive that should be going on during level load (except things like uncompressing from disk to make up for poor hard drive speeds). And probably random properties I guess. If you have 1000 monsters that start with random positions, that might take a while. But most games aren't like that.

Quote:

I just can't understand how people can store their assets in a format that was never meant to be a lossy image storage format. It was designed as a compression with a GPU-friendly decoding process. It is good at doing this for now, but why use it for something where other, better alternatives exist ?


If you're storing textures on disk, you probably want to avoid JPG because it's the worst of both worlds: artifacts and uncompressed in VRAM (or worse, you're DXT compressing the JPG, adding artifacts to artifacts). That leaves lossless, where PNG is king. Or hardware accelerated lossy, where DDS makes the most sense, since it natively supports it. Or roll your own, of course, that could support either.

If/when they invent hardware accelerated JPG and PNG compression, I will totally jump on that bandwagon. But I would still probably be using a DDS format: Microsoft would probably invent some new DDS mode that lets you store as JPG or PNG compression to keep DDS current. Compression aside, DDS is the best general purpose texture format, because it's the only one specifically developed with game hardware in mind.
Darwinbots - Artificial life simulation

#52 Yann L   Moderators   -  Reputation: 1794

Like
0Likes
Like

Posted 28 August 2008 - 10:01 PM

Quote:
Original post by Numsgil
How so? If the compressed, but larger resolution texture is the same size in memory as an uncompressed, smaller version. Also, I don't think it'll increase memory usage even counting the mip map levels (compared with an uncompressed but smaller texture + its mip maps). Or if it does, it's like 4 bytes for the one extra level :)

Sure, the difference is small - but saying that it is the same would be incorrect. Nitpicking, I admit ;) However, the increase of memory footprint on the cache is very much noticeable, since texture cache usually operates on decoded values.

Quote:
Original post by Numsgil
In your scenario you're doing exactly what I'm saying. You're essentially designing your game with imaginary future tech cards in mind, with large texture sizes, and allowing current lowest denominator cards to run by compressing the textures.

So ? Set aside that such cards aren't imaginary (but haven't yet reached the required market penetration for the target audience), my approach can seamlessly improve texture quality on such cards. Yours cannot without shipping a new texture pack.

Quote:

Really? Is it that you're afraid the artists won't ever choose the compression because they don't want their art to have artifacts?

Amongst others, that is a factor. Another one is that artists may not understand all technical implications of such a choice. We usually try to limit the options of our artists to 'art thingies' only.

Quote:

What else are you doing in your scene? The time it takes to dump stuff (textures, models, etc.) from disk to GPU should be the bottleneck. If other stuff is slowing your loading times down, I can't help but feel you're doing something wrong.

Lol, no, I don't think so ;)

Quote:

I can't imagine anything CPU intensive that should be going on during level load (except things like uncompressing from disk to make up for poor hard drive speeds).

Let's see - transcoding PRT coefficient maps for partially-dynamic global illumination, generating and compiling metashaders, tracing horizon maps, setting up dynamic partitioning systems, uncompressing progressive LOD data, preparing energy transfer volumes for dynamic soft shadows, generating seed and permutation tables for fractal vegetation and clouds, and so on. All this is dependent on the hardware the engine is being run on, in order to smoothly adjust the delicate performance-quality balance to match the players/users hardware and preferences. There's a lot more to a modern 3D engine than 'dumping stuff from disk to GPU'. Now granted, what we do is very high end stuff, and it's not for a game. But most of these techniques also apply to advanced game 3D engines.

Quote:

If/when they invent hardware accelerated JPG and PNG compression, I will totally jump on that bandwagon. But I would still probably be using a DDS format: Microsoft would probably invent some new DDS mode that lets you store as JPG or PNG compression to keep DDS current. Compression aside, DDS is the best general purpose texture format, because it's the only one specifically developed with game hardware in mind.

Let's not slip into religious fanboi-isms, and stay with the facts, shall we ? DDS is definitely not the best general purpose format. DDS isn't even a general purpose format. It is a quite specific format, that suits a certain type of situation, for certain people. Take console development, for example. Using precompressed DDS on XBox makes perfect sense, because your target hardware will never change.

There's nothing wrong with using DDS - but make sure you understand the implications of doing so, that you understand the available alternatives, and that after careful consideration, you have opted that using DDS is in fact the best solution in your situation.

But advocating it as a generalized 'best format', just because you happen to be familiar with it, without taking into account the specific context of usage, is very naive and misleading.

#53 Kest   Members   -  Reputation: 547

Like
0Likes
Like

Posted 28 August 2008 - 11:45 PM

Quote:
Original post by Hodgman
Quote:
Original post by Kest
Quote:
Also, while PNG might well be compressed it is only compressed on disk; DXT compressed images remain compressed in texture ram

Any format can be compressed once its in system ram.
IIRC, DirectX actually has the ability to load a PNG and save it as DDS, or vice versa. Once the texture is in ram, all format bets are off.
^^fixed to show apples and oranges ;)

If you're loading in one format and performing a conversion in system ram, then it increases your load texture processing times, which may decrease load times.

^^fixed to show apples and oranges ;)

You didn't fix my statement, you modified it. I was referring to texture ram. Once the texture is in video ram, it no longer matters how it was stored on the disk. When, where, and how it gets processed from disk to v-ram doesn't change any of the facts.

#54 frob   Moderators   -  Reputation: 19633

Like
0Likes
Like

Posted 29 August 2008 - 12:58 AM

Quote:
Original post by Numsgil
Quote:

Really? Is it that you're afraid the artists won't ever choose the compression because they don't want their art to have artifacts?

Amongst others, that is a factor. Another one is that artists may not understand all technical implications of such a choice. We usually try to limit the options of our artists to 'art thingies' only.
Ditto on that.

Sure, artists are able to pick the best format, it isn't that they will lack the skill if you give them the proper training. Some of them will understand the importance of picking the most appropriate format for the technical rather than artistic reasons, even if it reduces the artistic quality or image fidelity from the originals.

But the real problem is consistency.

They're dealing with many hundreds, even potentially several tens of thousands of graphics files over the course of the project. They're working with these files all day, every work day.

Do you honestly expect them to evaluate each file and determine the best format whenever they save their work? Every time they modify a file?

I don't. I expect a tool to do this when it builds and packages all the art assets. Automatic this task saves between hundreds and thousands of hours over the course of the project, depending on the number of artists. It simplifies their workflow, and it also gives better results for the game. Everybody wins.

In fact, we do have tools for this. The tool converts it into all the possible formats, evaluates them based on the resulting image and the data size, and picks the best one based on those results. The art programmers, technical artists, and other qualified people determine the heuristics for evaluating what is the best, allowing us to trade off size, speed, and most importantly image fidelity without sacrificing technical quality.



#55 Numsgil   Members   -  Reputation: 501

Like
0Likes
Like

Posted 29 August 2008 - 04:55 AM

Quote:
Original post by Yann L
Quote:

If/when they invent hardware accelerated JPG and PNG compression, I will totally jump on that bandwagon. But I would still probably be using a DDS format: Microsoft would probably invent some new DDS mode that lets you store as JPG or PNG compression to keep DDS current. Compression aside, DDS is the best general purpose texture format, because it's the only one specifically developed with game hardware in mind.

Let's not slip into religious fanboi-isms, and stay with the facts, shall we ? DDS is definitely not the best general purpose format. DDS isn't even a general purpose format. It is a quite specific format, that suits a certain type of situation, for certain people. Take console development, for example. Using precompressed DDS on XBox makes perfect sense, because your target hardware will never change.

There's nothing wrong with using DDS - but make sure you understand the implications of doing so, that you understand the available alternatives, and that after careful consideration, you have opted that using DDS is in fact the best solution in your situation.

But advocating it as a generalized 'best format', just because you happen to be familiar with it, without taking into account the specific context of usage, is very naive and misleading.


Best for game development I meant :) And not counting whatever you make yourself, of course. And compression aside (yes, that's a rather large aside). Other than compression there's nothing that PNG or TGA or JPG get you above what DDS gets you, while DDS gets you some things beyond PNG et al. Like native support for cubemaps. Sure you could just do a cross in a PNG, but why would you when DDS can natively support it? You know, besides that whole compression thing.

Unless I'm missing something?
Darwinbots - Artificial life simulation

#56 Numsgil   Members   -  Reputation: 501

Like
0Likes
Like

Posted 29 August 2008 - 05:06 AM

Quote:
Original post by frob
Quote:
Original post by Numsgil
Quote:

Really? Is it that you're afraid the artists won't ever choose the compression because they don't want their art to have artifacts?

Amongst others, that is a factor. Another one is that artists may not understand all technical implications of such a choice. We usually try to limit the options of our artists to 'art thingies' only.
Ditto on that.

Sure, artists are able to pick the best format, it isn't that they will lack the skill if you give them the proper training. Some of them will understand the importance of picking the most appropriate format for the technical rather than artistic reasons, even if it reduces the artistic quality or image fidelity from the originals.

But the real problem is consistency.

They're dealing with many hundreds, even potentially several tens of thousands of graphics files over the course of the project. They're working with these files all day, every work day.

Do you honestly expect them to evaluate each file and determine the best format whenever they save their work? Every time they modify a file?

I don't. I expect a tool to do this when it builds and packages all the art assets. Automatic this task saves between hundreds and thousands of hours over the course of the project, depending on the number of artists. It simplifies their workflow, and it also gives better results for the game. Everybody wins.

In fact, we do have tools for this. The tool converts it into all the possible formats, evaluates them based on the resulting image and the data size, and picks the best one based on those results. The art programmers, technical artists, and other qualified people determine the heuristics for evaluating what is the best, allowing us to trade off size, speed, and most importantly image fidelity without sacrificing technical quality.


I know this is something of a pipe dream, but seriously, artists should understand the technical implications of their work! This hand holding and coddling for artists that seems to be prevalent makes my stomach turn. At the very least an artists should have a basic grounding in calculus and linear algebra. They should understand the lighting model. They should be able to write their own shaders. They should understand and make decisions about how to use the hardware of video cards. I'm guessing this isn't the way things are simply because there aren't enough artists that can do that, and simple demand and lack of supply is driving the standards.

In a perfect world, I as a programmer would give a max budget to my art team. Do not exceed 160 MB. They would build their level with a mind towards that limit. If there's a far away billboard, they can save MBs by saving it as a DXT1. If there's some grass off in the distance, they might need an alpha channel and opt for DXT3 or 5. If there's a baddy that will be right in front of the player, they can go for a full 32 bits. Something in game counts the total MBs of all assets and gives it as a readback when the artists run the game. The art lead cracks down when some junior artist uses a 1024x1024 32 bit texture for a postage stamp. If render times are too high, they'd load up PerfHud and take a look at the draw commands.

I know that's not reality. I don't know if it's even achievable. Perhaps we should just be thankful when an artist makes something look good, and leave all the technical implications to the programmers.

#57 Zipster   Crossbones+   -  Reputation: 580

Like
0Likes
Like

Posted 29 August 2008 - 06:04 AM

Quote:
Original post by Numsgil
I know this is something of a pipe dream, but seriously, artists should understand the technical implications of their work! This hand holding and coddling for artists that seems to be prevalent makes my stomach turn. At the very least an artists should have a basic grounding in calculus and linear algebra. They should understand the lighting model. They should be able to write their own shaders. They should understand and make decisions about how to use the hardware of video cards. I'm guessing this isn't the way things are simply because there aren't enough artists that can do that, and simple demand and lack of supply is driving the standards.

Calculus and linear algebra? Pipe dream indeed ;) But seriously, what you're talking about sounds a lot like a technical artist, and there are plenty of those out there if you know where to look. We just hired one a few weeks ago. They're basically programmers who also have a good understanding of the art side of development as and can use Max, Photoshop, write scripts, shaders, etc. I don't know if I'd trust a full-blown artist to write shaders though. Shaders can truly be time-critical code if you're GPU-bound, and in my experience artists will use every single feature they possibly can to achieve a better visual effect unless you pull back on the reins a bit. We have a running joke in the graphics department whenever we add a new feature, "don't tell the artists!" :P

#58 Kest   Members   -  Reputation: 547

Like
0Likes
Like

Posted 29 August 2008 - 06:16 AM

Logic versus ingenuity. You can favor both equally, be naturally pulled toward one and annoyed with the other, or somewhere in between. To get the best of both worlds, you need at least two people. Or a split personality.

#59 Numsgil   Members   -  Reputation: 501

Like
0Likes
Like

Posted 29 August 2008 - 06:24 AM

Quote:
Original post by Zipster
Quote:
Original post by Numsgil
I know this is something of a pipe dream, but seriously, artists should understand the technical implications of their work! This hand holding and coddling for artists that seems to be prevalent makes my stomach turn. At the very least an artists should have a basic grounding in calculus and linear algebra. They should understand the lighting model. They should be able to write their own shaders. They should understand and make decisions about how to use the hardware of video cards. I'm guessing this isn't the way things are simply because there aren't enough artists that can do that, and simple demand and lack of supply is driving the standards.

Calculus and linear algebra? Pipe dream indeed ;) But seriously, what you're talking about sounds a lot like a technical artist, and there are plenty of those out there if you know where to look. We just hired one a few weeks ago. They're basically programmers who also have a good understanding of the art side of development as and can use Max, Photoshop, write scripts, shaders, etc. I don't know if I'd trust a full-blown artist to write shaders though. Shaders can truly be time-critical code if you're GPU-bound, and in my experience artists will use every single feature they possibly can to achieve a better visual effect unless you pull back on the reins a bit. We have a running joke in the graphics department whenever we add a new feature, "don't tell the artists!" :P


Ha, thanks, that was a good buzz word to search for. Yes I think that's what I'm talking about. Let's just train up a small army of technical artists and leave the regular variety to concept art :D
Darwinbots - Artificial life simulation

#60 emeyex   Members   -  Reputation: 378

Like
0Likes
Like

Posted 29 August 2008 - 07:02 AM

Quote:
Original post by Numsgil
...At the very least an artist should have a basic grounding in calculus and linear algebra...

That's the best line I've heard all year! The funniest part is that you said "at the very least". But seriously, if you do hear that Leonardo da Vinci is looking for a job, let me know, I'd love to hire him :)




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS