BC2_UNORM (aka DXT3) only gives 16 possible values for alpha so it's pretty poor for smooth gradients, dithering at compression time might help a little.

You're better off using BC3_UNORM (aka DXT5), it uses block compression on the alpha channel and is generally going to produce better results in most circumstances.

Thanks for the suggestion! Worked perfectly!