Turns out copy and paste code = bad
VertexArray.Tv = VertexArray.Tv = (float)SegmentReference.Side.Y_Offset / (float)Patch.Height;
VertexArray.Tv = VertexArray.Tv = VertexArray.Tv + ( (float)SegmentReference.Sector.Height / (float)Patch.Height );
2 and 3 are of course supposed to add on to the first two. Anyway I changed it to 0. Now everything except those damn unpegged textures are fixed, but those seem to be rare.
Here are some more screenshots. This time at 640x480 with linear filtering.