There is some very good discussion here, and I appreciate all of the contributions. There are a couple of points that I think are worth restating, or clarifying, so that people might understand more deeply.
With regard to which aspect of geometry the TBN is associated, the TBN exists for any point on the surface, not only at each face, and also not only at each vertex. It may be computed at each vertex, or each pixel, or once per face for that matter, depending on how it is to be used (vertex or fragment shader or just asset calculations on the CPU) or what is convenient, but strictly speaking it exists continuously across the surface.
With regard to the tangent direction, it is not in general true that it is defined to point in the direction of change of the u coordinate. A tangent vector can be any vector that is located in the tangent plane at a point on the surface (the tangent plane being the plane that is orthogonal to the local surface normal at that point, er, for a surface that is continuously differentiable). So, as Bacterius noted, it is true that there are infinitely many possible tangent space choices at any point on the surface.
It happens that computing the tangent along the curve of the u direction is quite convenient, and, since it aligns with texture coordinates, this choice fits very nicely with computer rendering use cases. Also, on a mesh, standardizing on this choice enables continuity of the tangent spaces across the entire mesh surface.
As for the bitangent (which is a more appropriate term than binormal, when dealing with surfaces), it is in fact just like the tangent in that it lies in the tangent plane. It is a tangent vector, just a different one than the T in TBN. In computer graphics the bitangent is often computed along the v direction, again in order to align with texture coordinate directions and provide for TBN continuity, but there is that issue of the u and v directions being nonorthogonal in model/world space (vs. tangent space), as discussed in prior posts. The terathon page linked above discusses this orthogonality issue with a workable solution, concisely.
I know many resources out there explicitly define the tangent to be computed along the u direction and bitangent to be computed along the v direction, but to understand deeply you should recognize that these choices are made merely to achieve the alignment with the texture directions with the happy bonus of TBN continuity. I hope this reply helps someone understand more deeply, if only a little.