I want to try DXT5 format for my normal maps to save some memory, but i have ugly artifacts showing and i was wandering is this to be expected with this compression or am i doing something wrong?

Saving my "normal" normal map in photoshop as DXT5:

and used info from here to reconstruct Z, so my pixel shader looks like this:

#pragma pack_matrix(row_major)
...
#if 1 // normal bumpmap

    float3 bump = txBump.Sample(samLinear, IN.TexCoord).xyz * 2.0f - 1.0f;

#else // dxt5

    float3 bump = txBump.Sample(samLinear, IN.TexCoord).wyy * 2.0f - 1.0f;

    bump.z = sqrt(1 - saturate(dot(bump.xy, bump.xy)));

#endif



    float3x3 tangentBasis = float3x3(normalize(IN.TangentW), normalize(IN.BinormalW), normalize(IN.NormalW));

    float3 n = normalize(mul(bump.xyz, tangentBasis));
...

Output with DXT5 (notice artifacts):

and with "normal" normal map (no artifacts):

Renormalize the normals in the shader.

They come out normalized when reconstructing Z from X and Y. There's no need to do any extra normalization.

In fact you can probably drop the normalize on the last line of that shader.

@MJP

Since you appear to be using D3D10 or D3D11...

I just started with dx11 couple of days ago.

If i save it as 3Dc and use same code as above i got:

How should i properly handle 3Dc format?

Any tool you would recommend to convert my normal maps to BC5?

...your DDS loader might bload it as BC5 if you save it as 3Dc format

IDK i am using DirectXTK DirectX::CreateDDSTextureFromFile to load my textures. How can i see which format is my texture once it is loaded?

@Hodgman

Instead of using that photoshop plugin, would you be interested in writing your own tool using a library?

Not sure what you mean? Any existing tool you would recommend to convert my textures?

EDIT:

I tried removing swizzle and sampling 3Dc normally:

//float3 bump = txBump.Sample(samLinear, IN.TexCoord).wyy * 2.0f - 1.0f;
float3 bump = txBump.Sample(samLinear, IN.TexCoord).xyz * 2.0f - 1.0f;
bump.z = sqrt(1 - saturate(dot(bump.xy, bump.xy)));

It is BC5 as you said:

This wiki page says that 3dc & bc5 are the same

No, i use same sampler for all textures, it would show on uncompressed normal map also.

D3D11_SAMPLER_DESC sampDesc;
    std::memset(&sampDesc, 0, sizeof(sampDesc));
    sampDesc.Filter = D3D11_FILTER_MIN_MAG_MIP_LINEAR;
    sampDesc.AddressU = D3D11_TEXTURE_ADDRESS_WRAP;
    sampDesc.AddressV = D3D11_TEXTURE_ADDRESS_WRAP;
    sampDesc.AddressW = D3D11_TEXTURE_ADDRESS_WRAP;
    sampDesc.ComparisonFunc = D3D11_COMPARISON_NEVER;
    sampDesc.MinLOD = 0;
    sampDesc.MaxLOD = D3D11_FLOAT32_MAX;
    hr = dev->CreateSamplerState(&sampDesc, &m_sampler);

3Dc looks fine for me for now.