My TBN (tangent-bitangent-normal) calculation works correctly for planar texture mapping and axis-aligned box mapping, including arbitrary rotations applied via a texture matrix. However, I'm getting an angular shift when I apply scaling to the texture coordinates and then try to calculate the TBN values.
My little brain can't seem to be able to chew through this - as best I can tell the distortion caused by scaling and box mapping (eg projection onto the primary plane) should be implicit in the coordinates themselves and should be handled naturally, but it isn't. The (currently per-face) TBN calculation is based on this and the code looks like this:
face->vNormal = VecNormalize(VecCrossProduct(vertices[face->indexes[0]] - vertices[face->indexes[1]], vertices[face->indexes[1]] - vertices[face->indexes[2]]));
IVector3D v0 = vertices[face->triindexes[0]];
IVector3D v1 = vertices[face->triindexes[1]];
IVector3D v2 = vertices[face->triindexes[2]];
IUVCoords2D uv0, uv1, uv2;
uv0 = face->uv0[0];
uv1 = face->uv0[1];
uv2 = face->uv0[2];
TReal x1 = v1.x - v0.x;
TReal x2 = v2.x - v0.x;
TReal y1 = v1.y - v0.y;
TReal y2 = v2.y - v0.y;
TReal z1 = v1.z - v0.z;
TReal z2 = v2.z - v0.z;
TReal s1 = uv1.u - uv0.u;
TReal s2 = uv2.u - uv0.u;
TReal t1 = uv1.v - uv0.v;
TReal t2 = uv2.v - uv0.v;
TReal r = 1.0f / (s1 * t2 - s2 * t1);
IVector3D tan0((t2 * x1 - t1 * x2) * r, (t2 * y1 - t1 * y2) * r, (t2 * z1 - t1 * z2) * r);
IVector3D tan1((s1 * x2 - s2 * x1) * r, (s1 * y2 - s2 * y1) * r, (s1 * z2 - s2 * z1) * r);
IVector3D n = face->vNormal;
face->vTangent = (tan0 - n * VecDotProduct(n, tan0)).Normalize();
//calculate handedness
face->vTangent.w = (VecDotProduct(VecCrossProduct(n, tan0), tan1) < 0.0f) ? -1.0f : 1.0f;
face->vBitangent = VecCrossProduct(face->vNormal.AsNormalized(), IVector3D(face->vTangent).AsNormalized());IEditablePolyFace* face = faces;
Since the texture looks exactly the way it should, I'm assuming the fault lies with the TBN calculation. I've attached a screenshot taken at an oblique angle of a wall with planar UV mapping. The texture has been arbitrarily rotated. The left image has vertical scaling of 1.4, the right image is unscaled. Red = normal, blue = tangent, green = bitangent.
Any ideas what might be wrong?