Josh Klint said:
I suppose you could construct the 3x3 matrix in such a way that the error is all packed into the tangent and bitangent

Actually you could get the tangent right in any case too? Just the bitangent would need an additional number for an eventual sheer?
That's still 5 numbers vs. 9.

However, i guess they made sure their UV maps have minimized stretching, and so they could ignore the error in practice.

Josh Klint said:
I don't think so, because there are many situations where the dot product of the tangent and the normal is not zero.

Yes, but only if artists (or tools) violate the ideal of UVs with minimized distortion, eventually on purpose.

This often makes sense, and eventually some users of your engine might think of enforced quaternions as a limitation. I assume it's rarely a problem in practice, but i do not really have experience with artwork on normal maps and can't tell. Ideally you would make it optional to the users, i guess.

But i do think alignment to normal and tangent works technically.
Even if they are not orthonormal, the projection of the tangent to the plane of texture can still match its desired direction exactly. So we get only a difference in magnitude, but afaik this would not matter since we only care about direction and normalize anyway.
To get the bitangent right as well, you could use a given signed sheer value like this: bitang = normalize (tangent.cross(normal) + tangent * sheer).

But maybe there are some special cases where magnitude matters, like anisotropic hair shaders or such stuff? Not entirely sure about that.

Edit: Maybe i think too much about texture space, ignoring the problem of mapping from that back to object space.
Related issues could be solved with compensating the normals in the texture map. But if those normals are available only in compressed formats, such preprocessing step would certainly cause a quality loss.