Any idea what's going on here?
DXT5 compresses the RGB channels together using one 'palette', and the Alpha channel independently using another, which means that the quantization/compression errors can be different in RGB than A.
If RGB gets a bit darker and A gets a bit lighter (or vice versa), then the final image after pre-multiplied alpha blending will be whacky, because the same translucency information is encoded in both, and has to stay in sync.
With traditional alpha blending, the alpha information is only in the Alpha channel, so any independent 'lossyness' in RGB won't affect it.
I have a feeling that if your DXT compressor knew that the content of the image was pre-multiplied alpha, it could do a better job. How do your DXT5 images get generated/compressed?
Pre-multiplied works fine with DXT1 because it's only got on/off alpha values, so they can't really "get out of sync". Maybe pre-multiplied and DXT5 simply aren't a good idea though?
the problem might go away if I had proper RGB values for the transparent parts, but I'm not sure how to make photoshop do that
I don't think this is your problem, but you make photoshop do this by painting RGB values in the transparent areas. Photoshop has two ways of dealing with transparency --
1) Don't use a "background" layer so that the background of your image is translucent.
2) Use a "background" layer so there are valid RGB values everywhere, but also add an alpha channel on the channels tab.
For web stuff, PNGs, etc, the first method works fine. For games, you should use the second method, and never use PNG as an intermediate format for translucent textures, as it throws out RGB values in transparent areas...