This looks all good. For the tetrahedron case the origin can be closest to a vertex, edge, face or be inside the tetrahedron. I would use Erin's approach where you compute the barycentric coordinates and use these to check the Voronoi regions. This is very structured and efficient.

Regarding the search direction it is all in the code. Use the right-hand-rule to figure it out.
http://en.wikipedia.org/wiki/Right-hand_rule

This looks all good. For the tetrahedron case the origin can be closest to a vertex, edge, face or be inside the tetrahedron. I would use Erin's approach where you compute the barycentric coordinates and use these to check the Voronoi regions. This is very structured and efficient.

Regarding the search direction it is all in the code. Use the right-hand-rule to figure it out.
http://en.wikipedia....Right-hand_rule

Alright so i've worked through it, and i have 2 more questions:

1) the normals for the faces of the triangles, im using the right hand rule, the problem is, how can i guarentee my normals face out when the arrangement of the simplex points may vary. For example in the figure below, those are respective cross products under each triangle to get outward facing normals. The ordering of the cross product varies depending on the arrangement of the simplex points. Im not sure how to proceed about guaranteeing that the normals i calculate point outwards.


2) What is a good way to test if the origin is closest to a triangular face, to test for. For example, lets say my simplex is the 'left' triangle above and the face pointing in the direction of ABxAC is being tested, my current code is:



if (m3dDotProduct3(ABxAC, AO) > 0) {
if (m3dDotProduct3(ACxAD, AO) <= 0) {
if (m3dDotProduct3(ADxAB, AO) <= 0) {
if (m3dDotProduct3(DCxDB, DO) <= 0) {
// origin is closest to ABC face
}
}
}
}




EDIT: okay so i put breakpoints in my different simplex solver cases. Every 'closest to a point' scenario except A for a line simplex has not been called once. I also put code to ensure my normals face outside the tetrahedron, however according to my break points the normals ALWAYS face outwards if i use the 'left' triangle from the image cross products.

My algorithm is having trouble with getting stuck in infinite loops... urghh

You can use the triple product and compute the volume when you are creating a tetrahedron simplex. If you get a negative result you can revert the winding of the triangle. Another way to verify this is to use the other vertex on a triangle. E.g. you looking at triangle ACD. Then you know that B must be behind this triangle. Same is true for all other combinations.

You can use the triple product and compute the volume when you are creating a tetrahedron simplex. If you get a negative result you can revert the winding of the triangle. Another way to verify this is to use the other vertex on a triangle. E.g. you looking at triangle ACD. Then you know that B must be behind this triangle. Same is true for all other combinations.

yeah turns out my issue isn't with my normals it's my tetrahedron edge and face tests they arent testing for the proper areas