Ok, everything I know about this matter comes from this Gamasutra article. To interpolate positions over the triangle, you use this formula:
To calculate two tangent vectors in order to calculate the surface normal. You need to use two partial derivatives of the above formula given as so:
How this looks in code for a 3rd order bezier triangle, or how I think this looks in code, can be seen in my domain shader.
#define p300 0
#define p030 1
#define p003 2
#define p210 3
#define p201 4
#define p120 5
#define p021 6
#define p012 7
#define p102 8
#define p111 9
#define U bary.x
#define V bary.y
#define W bary.z
cbuffer PerFrameBuffer : register(b0)
{
//float4x4 World; This is just the identity matrix, so not needed
float4x4 ViewProjection;
float4 vecEye;
float4 LightDirection;
float4 LightColor;
};
struct PatchTess
{
float EdgeTess[3] : SV_TessFactor;
float InsideTess : SV_InsideTessFactor;
};
struct HullOut
{
float3 PositionWorld : POSITION;
float2 MainTexCoord : TEXCOORD;
};
struct DomainOut
{
float4 PositionH : SV_Position;
float3 Normal : NORMAL0;
float3 Tangent : TANGENT0;
float3 Bitangent : BITANGENT0;
float3 View : NORMAL1;
float2 MainTexCoord : TEXCOORD0;
};
[domain("tri")]
DomainOut DS(PatchTess patchTess, float3 bary : SV_DomainLocation, const OutputPatch<HullOut,10> cp)
{
DomainOut dout;
float3 position =
cp[p300].PositionWorld * pow(U,3) +
cp[p030].PositionWorld * pow(V,3) +
cp[p003].PositionWorld * pow(W,3) +
cp[p210].PositionWorld * 3 * pow(U,2) * V +
cp[p201].PositionWorld * 3 * pow(U,2) * W +
cp[p120].PositionWorld * 3 * U * pow(V,2) +
cp[p021].PositionWorld * 3 * pow(V,2) * W +
cp[p012].PositionWorld * 3 * V * pow(W,2) +
cp[p102].PositionWorld * 3 * U * pow(W,2) +
cp[p111].PositionWorld * 6 * U * V * W;
float3 tangent =
cp[p300].PositionWorld * pow(U,2) +
cp[p120].PositionWorld * pow(V,2) +
cp[p102].PositionWorld * pow(W,2) +
cp[p201].PositionWorld * 2 * U * W +
cp[p210].PositionWorld * 2 * U * V +
cp[p111].PositionWorld * 2 * V * W;
float3 bitangent =
cp[p030].PositionWorld * pow(V,2) +
cp[p012].PositionWorld * pow(W,2) +
cp[p210].PositionWorld * pow(U,2) +
cp[p120].PositionWorld * 2 * U * V +
cp[p021].PositionWorld * 2 * V * W +
cp[p111].PositionWorld * 2 * U * W;
tangent = normalize(tangent);
bitangent = normalize(bitangent);
float3 normal = normalize(cross(tangent,bitangent));
dout.View = vecEye.xyz - position.xyz;
dout.PositionH = mul(float4(position,1.0f),ViewProjection);
dout.Normal = normal;
dout.Tangent = tangent;
dout.Bitangent = bitangent;
dout.MainTexCoord = cp[p300].MainTexCoord * U + cp[p030].MainTexCoord * V + cp[p003].MainTexCoord * W;
return dout;
}
However, the results I am getting are not quite right, and I wonder if I have some misunderstanding of how this is supposed to work. The positions calculated across each bezier triangle are exactly correct, but I am having trouble calculating a usable and continuous normal from the tangents. The following screenshot uses the above domain shader and is basically a series of instanced bezier triangles defining a height map drawn in two draw calls. One with triangles pointing down and another with triangles pointing up.
[attachment=14966:Bad Normals.jpg]
You can see here some strange effects, the directional light source comes from the top, and there is some sense that some triangles have proper lighting while others do not. I originally thought that the angle between the two tangents was unpredictable, so some of the normals were pointing the opposite direction, but if I brute force the normals to point up with the following code the situation is improved but still not right.
float3 normal = normalize(cross(tangent,bitangent));
if(normal.y < 0.0)
normal = normalize(cross(bitangent,tangent));
[attachment=14967:Bad Normals2.jpg]
Notice that each tri instance has one continuous edge with its neighbor, but the other two edges are wrong. Also I have no idea what causes the diagonal artifact in both images other than my calculations being just wrong. Ultimately I would like a heighmap made from bezier triangles with continuous normals across the whole field. Does anyone here have experience working with bezier triangles who can tell me where my assumptions are wrong?