Ok, I have the current distance based LOD scheme set up in my hull shader. It works pretty well, but has some non-optimal aspects. cp[0], cp[1] and cp[2] are the corners of my triangle patch.

PatchConstants PatchHS(InputPatch<VertexOutput,10> cp, uint patchID : SV_PrimitiveID)
{
	PatchConstants pt;

	float3 mid0 = lerp(cp[1].PositionWorld,cp[2].PositionWorld,0.5f);
	float3 mid1 = lerp(cp[2].PositionWorld,cp[0].PositionWorld,0.5f);
	float3 mid2 = lerp(cp[0].PositionWorld,cp[1].PositionWorld,0.5f);

	const float dMax = 250.0f;

	float d = distance(mid0,vecEye.xyz);
	pt.EdgeTess[0] = 12.0f * pow(saturate((dMax - d)/dMax),3);

	d = distance(mid1,vecEye.xyz);
	pt.EdgeTess[1] = 12.0f * pow(saturate((dMax - d)/dMax),3);

	d = distance(mid2,vecEye.xyz);
	pt.EdgeTess[2] = 12.0f * pow(saturate((dMax - d)/dMax),3);

	d = distance(cp[9].PositionWorld,vecEye.xyz);
	pt.InsideTess = 16.0f * pow(saturate((dMax - d)/dMax),2);

	return pt;
}

I'm using the midpoint of my triangle patch edges to calculate distance to the camera. Visually, the need for more or less triangles is nowhere near linear in it's relationship to distance, so I am also applying an exponential curve that sort-of, kind-of helps. This scheme is an improvement over simply using fixed tessellation factors, but it seems that it should be possible to come up with something better. The following screenshot shows one of the weaknesses of a distance based scheme.

[attachment=15203:TessFactors.jpg]

You can see a mountain in the foreground and a valley between this mountain and other mountains in the background. The tiles in the valley are both relatively flat and also oblique to the camera so that they need very little tessellation, yet they are being rendered with more triangles than the mountains in the background that could actually use some extra triangles. Wasteful in the former case and sort of ugly in the latter.

Before I tried the distance scheme, I tried to tessellate based on the screen length of the respective patch edges, but I couldn't figure out how to get it working. Such a scheme would be much more efficient and would also preserve water tight stitching since neighbor edges would have the exact same length. Does anybody know how I can calculate that in the hull shader? I tried multiplying my corner vertices by my ProjectionView matrix and then measuring length, but I couldn't figure out what to do with the results of that calculation. How long is a pixel in my world? I have no idea. If an edge goes from one corner of the screen to the other, how long is that? 100? 0.01? I couldnt find a useful context for the distance between my transformed corner control points.

Thanks for the reply Jason. I did my best to implement your screen space method, and it almost works except I have one or two bugs that seem impossible. It's one of those times I really wish I could debug my shader. Here is my new hull shader:

PatchConstants PatchHS(InputPatch<VertexOutput,10> cp, uint patchID : SV_PrimitiveID)
{
	PatchConstants pt;

	float4 corner0 = mul(float4(cp[0].PositionWorld,1.0f),ViewProjection);
	float4 corner1 = mul(float4(cp[1].PositionWorld,1.0f),ViewProjection);
	float4 corner2 = mul(float4(cp[2].PositionWorld,1.0f),ViewProjection);

	corner0 = corner0/(corner0.w + 0.00001f);
	corner1 = corner1/(corner1.w + 0.00001f);
	corner2 = corner2/(corner2.w + 0.00001f);

	const float dMax = 1.5f;

	float d0 = abs(distance(corner1.xy,corner2.xy));
	pt.EdgeTess[0] = 64.0f * saturate(d0/dMax);

	float d1 = abs(distance(corner2.xy,corner0.xy));
	pt.EdgeTess[1] = 64.0f * saturate(d1/dMax);

	float d2 = abs(distance(corner0.xy,corner1.xy));
	pt.EdgeTess[2] = 64.0f * saturate(d2/dMax);

	float d3 = min(d0,min(d1,d2));
	pt.InsideTess = 64.0f * saturate(d3/dMax);

	return pt;
}
 

I have two problems that I can see in the following screen shot that are probably related somehow. The most obvious problem is that no matter how much space a triangle takes up on screen, my inside tess factor is computing to 2 or less even though I'm clearly trying to use the smallest edge length for the calculation. The second problem I am having is that I can see that the bottom and left sides of the triangles are matching up with their neighbor, but the right side is not! How can only one side not match up? If I switched them there should be two sides not matching up. It would seem that my d2 value is bad or NaN or something, but I can't imagine why that would be, also it seems to almost be right, while NaN values usually jump out at you.

[attachment=15215:TessProb.png]

I almost wonder if my code is being optimized away or something. Here is a picture of the resulting gaps. You can see they are only on the one diagonal.

[attachment=15217:TessProb2.jpg]

So, do you have multiple LODs at the same time on screen ?

if so, did you consider you have to connect them via some intermediary patch that has High detail on one side and low on the other one ? Hunting for those bugs is no fun, though...

LOD in this sense only applies to the edge. There is no problem for each edge to have a different tess factor as long as they line up with their neighbors. There's no need for an intermediary patch.

On another note, when I simply use pt.EdgeTess[2] = 4.0, the gaps go away and the opposing triangles line up on all sides. That means that for some reason, the distance calculation is giving different results going backwards or forwards, e.i. the distance from corner0 to corner1 is different than the distance from corner1 to corner0. I can't imagine how it ends up that way. For the other edges everything lines up!

I went ahead and lowered the max tessellation from 64 to 12 in order to make the problem more obvious. Another thing that is really stumping me is that the inside tessellation result makes no sense at all, even with the problem on edge 2.

[attachment=15220:TessProb3.jpg]

I used this code to change it, and it works much more as intended: (Note that changed d0,d1 etc. to dist0, dist1 in case I was using some register name by accident)

	float dist3 = min(dist0,min(dist1,dist2));
//	pt.InsideTess = 12.0f * saturate(dist3/dMax);
	pt.InsideTess = min(pt.EdgeTess[0],min(pt.EdgeTess[1],pt.EdgeTess[2]));

But I can't imagine why this line would give a different result than the line commented out line! The edge factors are calculated the same way as the inside factor! If the bad edge is wrong but close to the others, why would the inside edge be zeroed out when I am using the same distance? The inside factor should match one of the three edges, and the commented out line it matches none of them.

Here is another observation that I did not expect. On the triangle patch in the center of the image, even though the edge nearest to the camera is nearly twice as long as the other two edges, all the edges appear to have the same tess factor! I know I can set them independently, so the three edges must be calculating to the same clip space length. That also explains why one edge is not lining up with its neighbor. It looks like my clip space calculations are wrong, because the distance between the edge points are always the same which is not the case.

Actually this consequence is visible in the above example as well. All of the edges of a single triangle have the same tess factor. What am I doing wrong?

EDIT: wrong again actually. Only the 1 and 2 edges are always the same, the 0 edge appears to be independent. I have no idea what's going on.

[attachment=15221:TessProb4.jpg]