It''s too late for me to think too clearly, but since when did that stop me The only comment I can really make at this point is that being in the plane or in a plane parallel is not two seperate cases. You calculate the closest point exactly the same way in both of them. You could form three planes using the normal of the plane containing the triangle and each of the sides of the triangle. That would let you project the triangle into a parallel plane that contains your vector. The closest point to that triangle is also the closest point to the original triangle. So there is no differance between the two.

One thought in the more general case is that you could draw two lines perpendicular to each side of the triangle positioned at each end of the side. If a point is between two lines for a side then it''s distance from the triangle is the distance from that side. If it is outside all of them then its distance is the distance from a vertex and which vertex it is is the one you can get to without crossing any lines. If it wasn''t so late I might actually have some suggestions on how you could use that for this purpose.

I wrote some code to do this a while ago, as part of some test code for a v-clip implementation. The goal was to be able to find the precise distance between polyhedron features as v-clip iterated through them, to check it was converging. The code is robust but not at all fast.

I first coded a number of distance routines for points, lines and polygons. These were used to work out the distance between features.

So for a triangle and edge I have an edge-edge distance function to find the shortest distance from the edge to the edges of the triangle. I also do further checks in case either the endpoints are nearer to the triangle face (if so they muat lie in the prism it is a cross section of) or in case the edge crosses the triangle (so the distance is zero).

If you need the point you will need to keep track of it a you go along, e.g. work out the point that is nearest at each step and if the distance is shortest remember the point.

I would be careful of dealing with cases where multiple points are nearest, as for a random 3D line and plane this should be impossible, e.g. it could happen but the probability is zero. In practice it will happen, but very rarely, and it will happen at the point where the limit of floating point accuracy will be significant, making a choice of ''best'' point pretty arbitrary.