Hi I have two triangles sharing an edge and I have their normal vectors.
Na Nb \ F / \ /|\ / \/ | \/ /\ | /\ \ | / \ | / \|/ G
(Na and Nb are the triangle normals, F and G are their shared edge)
I want to check if the angle between the normals is: = 0, < PI, = PI or > PI.
I'm a bit rusty on 3d related math, but I looked at dot product and I don't think that works since I believe it can only be used to find the angle between 0 and PI, where as I want between 0 and 2*PI.
I think maybe something could be done with cross product, e.g. since normals are unit length I have:
Na * Nb = sinX*V (V being the cross product, X being the angle).
But I don't know where to go from there.
I've looked at this post from a similar/same question: http://www.gamedev.net/topic/112385-detecting-if-2-triangles-are-concave-or-convex/?view=findpost&p=1604381
I presume A and B are supposed to indicate the surface normals? I''ll ut the non-orthogonality of the pictures down to the limitations of ascii art!
There may indeed be an easier method, but here is one answer to your problem. First, impose an ordering of traversal along the edge. Call A the first triangle edge and B the second triangle edge.
1) Compute the tangent vector (T) to B, so that T.NB = 0. Additionally, choose the positive direction for the tangent vector to be away from A (i.e., in the positive traversal direction)
2) Compute the components of the normal NA in the directions of T and NB, so that NA = a1T + a2NB
A is concave if a1 < 0 and convex if a1 > 0.
As I said though, there may be an easier way... I just cannot think of it at this time.
But in this I don't understand what the a1 and a2 is (I'm guessing something like sinX), or what he means on the second step, compute the components?