Hi I have two triangles sharing an edge and I have their normal vectors.

Na          Nb
 \    F    /
  \  /|\  /
   \/ | \/
   /\ | /\
   \  |  /
    \ | /
     \|/
      G

(Na and Nb are the triangle normals, F and G are their shared edge)

I want to check if the angle between the normals is: = 0, < PI, = PI or > PI.

I'm a bit rusty on 3d related math, but I looked at dot product and I don't think that works since I believe it can only be used to find the angle between 0 and PI, where as I want between 0 and 2*PI.

I think maybe something could be done with cross product, e.g. since normals are unit length I have:

Na * Nb = sinX*V (V being the cross product, X being the angle).

But I don't know where to go from there.

I've looked at this post from a similar/same question: http://www.gamedev.net/topic/112385-detecting-if-2-triangles-are-concave-or-convex/?view=findpost&p=1604381

I presume A and B are supposed to indicate the surface normals? I''ll ut the non-orthogonality of the pictures down to the limitations of ascii art!

There may indeed be an easier method, but here is one answer to your problem. First, impose an ordering of traversal along the edge. Call A the first triangle edge and B the second triangle edge.

1) Compute the tangent vector (T) to B, so that T.NB = 0. Additionally, choose the positive direction for the tangent vector to be away from A (i.e., in the positive traversal direction)

2) Compute the components of the normal NA in the directions of T and NB, so that NA = a1T + a2NB

A is concave if a1 < 0 and convex if a1 > 0.

As I said though, there may be an easier way... I just cannot think of it at this time.

Cheers,

Timkin

But in this I don't understand what the a1 and a2 is (I'm guessing something like sinX), or what he means on the second step, compute the components?

Thanks.

(Na and Nb are the triangle normals, F and G are their shared edge)
I want to check if the angle between the normals is: = 0, < PI, = PI or > PI.

The limitation of using the dot product is that you can't work out the direction of the fold. 90 degrees either way looks the same. The trick would be to use some other technique to work out the fold direction. I can't think of a particularly elegant way to calculate the direction: One idea is to take the normal of your first triangle and dot product it with one of the edge directions from your second triangle (just not the shared edge!). The result will be a positive or negative number depending on which direction the fold is.

Yes this was the problem I ran into, I'll give your suggestion a try.

What's an example where the angle is more than PI?

The triangles are part of a 3D mesh, so I'm trying to tell if they are concave or convex, so >= PI is convex and < PI is concave.

The vertices are winded clockwise so the triangle normals always on the outer surface of the mesh.

EDIT:

Ah I just realised what you meant, I've been looking at the problem a bit wrong. I'll have think about this a bit more and get back to you.

Okay, after Álvaro's question I realised I was confusing the problem. for two triangles sharing an edge I want to find out whether they are facing away from each other, facing towards each other or laying flat (same normal).

So a slight change from C0lumbo's method, I dot product the normal from the first triangle to the non shared edge of the second triangle, where dotproduct=0 is both triangles laying flat, dotproduct<0 facing away (convex), dotproduct>0 facing towards (concave).

Seems like it should work.

Thanks, I'll give it a try when I get time.

It looks to me like you are both conceptually doing the same thing: getting rid of your angular discrepancy problems with cos(theta) by rotating the problem space. Alvaro's answer is accomplishing this by explicitly computing the third axis of one of the triangles... whereas CaptainMurphy's answer is getting a vector pointed in the same general direction by simply using one of the edges.

These two answers will produce the same (correct) result... at least insofar as they both return positive or negative for the same inputs under an assumption of shared edge GF. Alvaro's answer doesn't depend on the shared-edge asumption, but under this assumption CaptainMurphy's answer is probably a little faster.