I went ahead and lowered the max tessellation from 64 to 12 in order to make the problem more obvious. Another thing that is really stumping me is that the inside tessellation result makes no sense at all, even with the problem on edge 2.
I used this code to change it, and it works much more as intended: (Note that changed d0,d1 etc. to dist0, dist1 in case I was using some register name by accident)
float dist3 = min(dist0,min(dist1,dist2)); // pt.InsideTess = 12.0f * saturate(dist3/dMax); pt.InsideTess = min(pt.EdgeTess,min(pt.EdgeTess,pt.EdgeTess));
But I can't imagine why this line would give a different result than the line commented out line! The edge factors are calculated the same way as the inside factor! If the bad edge is wrong but close to the others, why would the inside edge be zeroed out when I am using the same distance? The inside factor should match one of the three edges, and the commented out line it matches none of them.