Jump to content

  • Log In with Google      Sign In   
  • Create Account


Why do game companies use custom texture formats instead of the DDS format?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
85 replies to this topic

#21 Yann L   Moderators   -  Reputation: 1794

Like
0Likes
Like

Posted 27 August 2008 - 02:57 AM

Quote:
Original post by Numsgil
Having worked with DDSs, I would strongly recommend them to anyone unfamiliar with them. You can reduce the memory footprint (in VRAM I mean) by 4 to 8 times (depending on which DXT format you're using). Which means you can cram more or larger textures into the card, and get better quality results.

While this is true, it has nothing to do with the DDS format. You can obviously also load uncompressed data from a PNG, or some other custom format, and compress it on the fly. That's what we do. It has the advantage that compression can be turned on and off by a user quality preference setting, and that it seamlessly supports other forms of existing or upcoming compression without rebuilding all the assets.

And of course, if we talk about lossless compression, PNG is lightyears ahead of DDS.


Sponsor:

#22 Wyrframe   Members   -  Reputation: 731

Like
0Likes
Like

Posted 27 August 2008 - 03:09 AM

The points to using DDS...

  • Pre-compression: compress your textures at production time, so you can see exactly what compression artifacts will result. Pick the compression algorithm at production time to get better results, for your own definition of "better".
  • Mipmap pre-generation: if you don't like how a given mipmap level looks, replace it with a tweaked version to make it sharper, or fuzzier, or what have you. You can also regenerate the higher mipmapping levels from the newly inserted mipmap, instead of from the base level.
  • Direct DX#Surface support: generate cubemaps, 3D textures, all kinds of specialized surfaces, all from the texture resources in the DDS file, with a single call. Why re-invent the wheel?


You can still use your own archive format, as the loader still supports being given a block of memory and reading it as though it were a DDS file.

#23 EmptyVoid   Banned   -  Reputation: 99

Like
0Likes
Like

Posted 27 August 2008 - 04:05 AM

The thing is I already have a format that supports all my needs and it's just a matter of copying and pasting the code. I'm just wondering if there is any reason not to use it?

#24 jbadams   Senior Staff   -  Reputation: 18192

Like
0Likes
Like

Posted 27 August 2008 - 04:19 AM

Don't waste too much time second guessing such options.

If you've got something that meets all your needs then you may as well be using it rather than looking for problems.

If you've done your research well and it really does meet all your needs then you'll be fine. If not, then you'll quickly learn to do better initial research (not to mention properly figuring out your needs) in future.

As long as you're sure your needs are met don't worry about the reasoning for why other people (or companies) use or do not use a particular technology. A lot of the requirements and restrictions of professional development do not neccesarily apply to hobbyist work, and even when they do, the decisions made by professionals are not always the best ones.

#25 Kwizatz   GDNet+   -  Reputation: 1193

Like
0Likes
Like

Posted 27 August 2008 - 04:29 AM

Quote:
Original post by riruilo
Does anyone remember this library?


Either PhysFS or zziplib.


#26 ndatxcod   Members   -  Reputation: 100

Like
0Likes
Like

Posted 27 August 2008 - 04:48 AM

I really hate when people are talking about protecting data and someone ALWAYS comes up and says a nonsense along the lines of "you shouldn't bother protecting your data because if someone really wants to get it he will be successful anyway" it is usually spewed out by people who have never finished a commercial project by themselves. The point of protecting data is not making it impossible to access but to deter most users wanting to take a peek at it, or even to steal it for their own projects, also that data comes usually from third party vendors (sounds and music) who require you to protect it before shipping the product. Should you use your own format for your game ? If you intent to go pro, then yes, there are a lot of successful indies that are doing that and have their own file format for their games and I'm not talking about a simple password protected zip.

Now when I say file format I'm not referring to the "primitive" types like a png or a dds. I'm talking about your own format wrapping those common ones, sort of like a pak system where you can pack your data in a single file, you put sounds, images, maps, etc using your own set of functions. To read/steal it someone would have to disassemble the binary.

#27 ernow   Members   -  Reputation: 728

Like
0Likes
Like

Posted 27 August 2008 - 04:54 AM

There is a major difference between protecting people from tampering with the graphics and preventing them from stealing the graphics.

Encryption might slowdown an application but adding and checking a digital signature is not that expensive.

In the end, anything digital can be copied but you can make it hard. I have yet to find a image format that contains a digital signature, so that would be a reason to invent your own format.



#28 frob   Moderators   -  Reputation: 20495

Like
0Likes
Like

Posted 27 August 2008 - 09:28 AM

Quote:
Original post by ndatxcod
I really hate when people are talking about protecting data and someone ALWAYS comes up and says a nonsense along the lines of "you shouldn't bother protecting your data because if someone really wants to get it he will be successful anyway" it is usually spewed out by people who have never finished a commercial project by themselves. The point of protecting data is not making it impossible to access but to deter most users wanting to take a peek at it, or even to steal it for their own projects, also that data comes usually from third party vendors (sounds and music) who require you to protect it before shipping the product. Should you use your own format for your game ? If you intent to go pro, then yes, there are a lot of successful indies that are doing that and have their own file format for their games and I'm not talking about a simple password protected zip.

Now when I say file format I'm not referring to the "primitive" types like a png or a dds. I'm talking about your own format wrapping those common ones, sort of like a pak system where you can pack your data in a single file, you put sounds, images, maps, etc using your own set of functions. To read/steal it someone would have to disassemble the binary.

I've written my own homebrew games and sold them for profit. I've worked at independent professional software studios, and currently work for a major game corporation.

I'm going to repeat this line, only because it is true: Do not try to obscure, encrypt, or otherwise hide your data. There is no point to it. The data is virtually useless to anybody, and if they did manage to steal it, copyright law still protects it. If they modify it, it isn't a big deal, both because of copyright protection so we can take them to court if it's bad, and if it is neutral or good it amounts to word-of-mouth advertising.

I have always used my own custom data formats for countless projects, but it is absolutely not for reasons of protecting data.

So why do it?

The file formats are useful for exchanging data between programs for compatibility, but I'm not interested in that. Instead, I know the memory formats and layouts of my data as used in the final game. I use my tools to load the data to file in exactly the same format and layout that it is needed when running. When I need to load a level, I can read a huge block of data for the assets. I don't have to worry about parsing what is images and textures, what is models, what is transformation data, or otherwise manipulate or manage the data at all. I just read the type of data, it's length, and dump the entire thing into memory all at once. This cuts load times by HUGE amounts.

If it is compressed data, then it is a simple matter to decompressed the block as it is loaded, then dump the decompressed block directly into memory. Again, it loads more than just a single bit of data but a huge swath of memory. It just adds a single step to the loader, and generally helps the load time by a few percent.

Just to show the difference, when we started one project at a studio a few years back, we loaded and parsed all the asset files directly, without preprocessing them into a tool. The load time was around four or five minutes, while listening to the disk churning as each tiny file was loaded and parsed. We finally got the tool in place to preprocess the data and pack it up, and load time dropped to just a few seconds.

Load time, not protection of data, is the primary reason to pack data in custom formats.

#29 MatsVed   Members   -  Reputation: 100

Like
0Likes
Like

Posted 27 August 2008 - 10:01 AM

Quote:
If data obfuscation was the goal, we would see a whole lot more games that encrypted the hell out of everything, which we just don't see.


There's normal archives, and then there's Mike O'Brien Pack [grin]

#30 Kylotan   Moderators   -  Reputation: 3338

Like
0Likes
Like

Posted 27 August 2008 - 11:13 PM

Quote:
Original post by frob
Quote:
Original post by ndatxcod
The point of protecting data is not making it impossible to access but to deter most users wanting to take a peek at it, or even to steal it for their own projects, also that data comes usually from third party vendors (sounds and music) who require you to protect it before shipping the product.

I'm going to repeat this line, only because it is true: Do not try to obscure, encrypt, or otherwise hide your data. There is no point to it.

Frob, did you read his post? I've selectively quoted the important part. It is required. It may be practically pointless, but it doesn't make it any less necessary.



#31 frob   Moderators   -  Reputation: 20495

Like
0Likes
Like

Posted 28 August 2008 - 12:23 AM

Quote:
Original post by Kylotan
Quote:
Original post by frob
Quote:
Original post by ndatxcod
The point of protecting data is not making it impossible to access but to deter most users wanting to take a peek at it, or even to steal it for their own projects, also that data comes usually from third party vendors (sounds and music) who require you to protect it before shipping the product.

I'm going to repeat this line, only because it is true: Do not try to obscure, encrypt, or otherwise hide your data. There is no point to it.

Frob, did you read his post? I've selectively quoted the important part. It is required. It may be practically pointless, but it doesn't make it any less necessary.
No, I don't need to reread it. I saw that and hoped it was addressed clearly enough.

I've worked with several companies that had the requirement. Their goal has been to simply remove it from common container formats so people don't take it.

Every one of them have been satisfied by simply packing the data in something other than .wav, .mp3, .tga, .mb, or whatever other 'original' format the stuff took.

Since that is exactly what we are doing, repackaging it for our memory needs rather than their portable formats, it tends to fill the requirement.

#32 Kest   Members   -  Reputation: 547

Like
0Likes
Like

Posted 28 August 2008 - 02:59 AM

On the protection front, has anyone considered that obscuring/hiding data may be beneficial to players? Letting them easily browse through your assets is essentially like allowing theater play-goers to see your actors dressing up for their parts.

Everyone is curious, but not everyone really wants to see.

As for the OP question, I don't use the DDS format because it's not as widely supported by the various software I use for development. The PNG format is lossless, compressed, supports transparency, and is supported by every application I've come across. It's been ideal. My game stores all resource data, including textures, into a single game data file. There's no encryption on it, but one would need to write specialized software to open it.

#33 EmptyVoid   Banned   -  Reputation: 99

Like
0Likes
Like

Posted 28 August 2008 - 03:25 AM

So just wondering, does the png format support volume textures and cubemaps?

#34 swiftcoder   Senior Moderators   -  Reputation: 9882

Like
0Likes
Like

Posted 28 August 2008 - 03:44 AM

Quote:
Original post by EmptyVoid
So just wondering, does the png format support volume textures and cubemaps?
Volume textures no. Cubemaps can be trivially stored in any image format by storing by layting out the sides next to each other (typically in a cross shape), ala SOIL, Horde, etc.

#35 frob   Moderators   -  Reputation: 20495

Like
0Likes
Like

Posted 28 August 2008 - 07:54 AM

Quote:
Original post by Kest
On the protection front, has anyone considered that obscuring/hiding data may be beneficial to players? Letting them easily browse through your assets is essentially like allowing theater play-goers to see your actors dressing up for their parts.

Everyone is curious, but not everyone really wants to see.

Quite a few games do provide them in easy formats.

Many games, even from megacorps and major publishers, have audio files directly in mp3, wma, ogg, or other formats, ready for your listening pleasure. If you look on console game disks, you can sometimes find these.

Occasionally you can find cut scenes in wma, swf, or mpg formats. Some have .bik and .smk files, which are less obviously movies for the uninitiated.

In homebrew and F/OSS games you see games that include a ton of image files, but they are very rare in larger commercial games, for reasons discussed above.


#36 phantom   Moderators   -  Reputation: 7155

Like
0Likes
Like

Posted 28 August 2008 - 07:59 AM

Quote:
Original post by Kest
As for the OP question, I don't use the DDS format because it's not as widely supported by the various software I use for development. The PNG format is lossless, compressed, supports transparency, and is supported by every application I've come across. It's been ideal. My game stores all resource data, including textures, into a single game data file. There's no encryption on it, but one would need to write specialized software to open it.


What applications don't support DDS then?

Also, while PNG might well be compressed it is only compressed on disk; DXT compressed images remain compressed in texture ram, which is basically 'free' AND has the added benifit of cutting down bandwidth requirements for texture sampling as well as making better use of the texture cache on the chips (again, removing bandwidth pressure). For many images, even if the compression is lossy, this doesn't matter as you simply don't notice.

Granted, not everything needs to be compressed (much like you shouldn't turn on mipmaping and AF filtering by default for all textures) but it can be a good thing to take advantage of.

#37 Yann L   Moderators   -  Reputation: 1794

Like
0Likes
Like

Posted 28 August 2008 - 08:25 AM

Quote:
Original post by phantom
Also, while PNG might well be compressed it is only compressed on disk; DXT compressed images remain compressed in texture ram, which is basically 'free' AND has the added benifit of cutting down bandwidth requirements for texture sampling as well as making better use of the texture cache on the chips (again, removing bandwidth pressure). For many images, even if the compression is lossy, this doesn't matter as you simply don't notice.

As I said somewhere above, it doesn't make sense to compare on-disk storage formats and in-VRAM formats, since they can be converted on the fly. And it's not slower either. We have a threaded system that will load and decode a (container-wrapped) PNG on one core, while at the same time encoding and uploading the previous DXT texture on the second core. The current bottleneck of the system is the harddisk. In fact, it is even faster than the loading of precompressed DDS, since PNGs will often require less harddisk IO to load. And we have the added benefit of letting the user choose a quality compression level in the preferences (eg. when he switches from a 512MB to a 1GB VRAM graphics card...)

Oh, and btw, of course PNG can store volume textures, texture arrays, cubemaps and mipmaps.

#38 swiftcoder   Senior Moderators   -  Reputation: 9882

Like
0Likes
Like

Posted 28 August 2008 - 08:51 AM

Quote:
Original post by phantom
What applications don't support DDS then?
Pretty much anything on a Mac [smile]
Seriously, there is exactly one external tool (Aorta) which produces DDS on the Mac.

Quote:
Original post by Yann L
Oh, and btw, of course PNG can store volume textures, texture arrays, cubemaps and mipmaps.
For some definition of store. Yes, you can store pretty much any aribtrary data in a PNG wrapper, but it doesn't provide anything much in the way of support for volume textures or mipmaps - i.e. no metadata (depth for volume textures, mipmap count for mipmaps, etc.).

#39 Numsgil   Members   -  Reputation: 501

Like
0Likes
Like

Posted 28 August 2008 - 09:05 AM

Quote:
Original post by Yann L
Quote:
Original post by phantom
Also, while PNG might well be compressed it is only compressed on disk; DXT compressed images remain compressed in texture ram, which is basically 'free' AND has the added benifit of cutting down bandwidth requirements for texture sampling as well as making better use of the texture cache on the chips (again, removing bandwidth pressure). For many images, even if the compression is lossy, this doesn't matter as you simply don't notice.

As I said somewhere above, it doesn't make sense to compare on-disk storage formats and in-VRAM formats, since they can be converted on the fly. And it's not slower either. We have a threaded system that will load and decode a (container-wrapped) PNG on one core, while at the same time encoding and uploading the previous DXT texture on the second core. The current bottleneck of the system is the harddisk. In fact, it is even faster than the loading of precompressed DDS, since PNGs will often require less harddisk IO to load. And we have the added benefit of letting the user choose a quality compression level in the preferences (eg. when he switches from a 512MB to a 1GB VRAM graphics card...)

Oh, and btw, of course PNG can store volume textures, texture arrays, cubemaps and mipmaps.


You should try compressing the DDSs on disk. So you take your texture, store it as a DDS using a lossy compression, and then compress that image using some lossless compression algorithm. It sounds weird but it should improve load times.
Darwinbots - Artificial life simulation

#40 Yann L   Moderators   -  Reputation: 1794

Like
0Likes
Like

Posted 28 August 2008 - 09:57 AM

Quote:
Original post by swiftcoder
Quote:
Original post by Yann L
Oh, and btw, of course PNG can store volume textures, texture arrays, cubemaps and mipmaps.
For some definition of store. Yes, you can store pretty much any aribtrary data in a PNG wrapper, but it doesn't provide anything much in the way of support for volume textures or mipmaps - i.e. no metadata (depth for volume textures, mipmap count for mipmaps, etc.).

PNG supports meta data chunks. We use them very successfully to store everything I listed above, and more. The only thing PNG can't be used for is HDR/fp textures (at least not without major modifications of the format). We designed special Photoshop and Maya plugins that transparently handle the volume/cubemap/mipmap data directly without user intervention.

Quote:
Original post by Numsgil
You should try compressing the DDSs on disk. So you take your texture, store it as a DDS using a lossy compression, and then compress that image using some lossless compression algorithm. It sounds weird but it should improve load times.

We don't want our on-disk assets to be lossy compressed. Our engine is used for highend architectural visualization, and customers often complain about DXTC artifacts that are visible on smooth and subtile colour gradients, as they are used in modern interior design. So we need the option to turn off DXT compression without changing the assets.

Besides that, I'm not sure how DXTC would actually compress. DXTC is a pretty crude compression system design for speed rather than quality. It generates a lot of quantization errors that could limit the way a lossless compressor may process the data. But I never tried it. Could be interesting to do.




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS