I'm normally a big fan of pre-multiplied alpha, but I'm having some issues with using it with a certain texture.

Here's a screenshot to demonstrate the problem:

http://mtnphil.files.wordpress.com/2013/03/example1.png

The artifacts are obvious in the top image. If I don't compress the texture, I get the image on the bottom, which looks good.

In the middle is what happens if I use non-premultiplied alpha and DXT5. It looks pretty good (still a few small artifacts). I'm surprised it is so much better than pre-multiplied alpha.

Here is what the RGB and A channels look like for the DXT-compressed texture (grabbed from PIX):

http://mtnphil.files.wordpress.com/2013/03/dxt5_texture_in_pix.png

It looks as I would expect.

Other notes:

- I also tried DXT3, but there wasn't much difference in quality.

- No mipmaps are involved. All textures are drawn 1-to-1 texel-to-pixel.

- All screenshots are 3x.

Any idea what's going on here? I have a feeling the problem might go away if I had proper RGB values for the transparent parts, but I'm not sure how to make photoshop do that. (and if that were the problem, why would the non-premultiplied DXT scenario look fine?).

Any idea what's going on here?

DXT5 compresses the RGB channels together using one 'palette', and the Alpha channel independently using another, which means that the quantization/compression errors can be different in RGB than A.

If RGB gets a bit darker and A gets a bit lighter (or vice versa), then the final image after pre-multiplied alpha blending will be whacky, because the same translucency information is encoded in both, and has to stay in sync.

With traditional alpha blending, the alpha information is only in the Alpha channel, so any independent 'lossyness' in RGB won't affect it.

I have a feeling that if your DXT compressor knew that the content of the image was pre-multiplied alpha, it could do a better job. How do your DXT5 images get generated/compressed?

Pre-multiplied works fine with DXT1 because it's only got on/off alpha values, so they can't really "get out of sync". Maybe pre-multiplied and DXT5 simply aren't a good idea though?

the problem might go away if I had proper RGB values for the transparent parts, but I'm not sure how to make photoshop do that

I don't think this is your problem, but you make photoshop do this by painting RGB values in the transparent areas. Photoshop has two ways of dealing with transparency --

1) Don't use a "background" layer so that the background of your image is translucent.

2) Use a "background" layer so there are valid RGB values everywhere, but also add an alpha channel on the channels tab.

For web stuff, PNGs, etc, the first method works fine. For games, you should use the second method, and never use PNG as an intermediate format for translucent textures, as it throws out RGB values in transparent areas...

I have a feeling that if your DXT compressor knew that the content of the image was pre-multiplied alpha, it could do a better job. How do your DXT5 images get generated/compressed?

I was using the default DXT compression in the XNA content pipeline. I also just tried using the nvidia photoshop plugin and saving directly as DXT5 in dds format (and I checked "pre-modulate color", which appears to save as pre-multiplied). I got pretty much the same result.

I think it just looks especially bad in this scenario because the background I'm drawing on top of is almost the same shader. If it's much darker or lighter then the artifacts don't show up so much.

So I doubt there's much I can do to make it look better if I'm using DXT.

2) Use a "background" layer so there are valid RGB values everywhere, but also add an alpha channel on the channels tab.

Ok, I tried that, and saved as DDS to preserve the RGB values. And you were right, this is not my problem.