Part of finding the shortest distance to the line involves finding the closest point on the line.

A quick google turned up this post: http://www.gamedev.net/topic/444154-closest-point-on-a-line/

if you have a normalized direction vector.

and a point, let's say:

Dir Vector = (1.0, 0.0, 0.0)

Point = (20.0, 10.0, 0.0)

the closest point on the line would be: (Dir.x*Point.x, Dir.y*Point.y, Dir.z*Point.z) (20.0, 0.0, 0.0)

let's say he have an arbitrary direction:

Dir Vector = (0.707, 0.707, 0.0)

Point = (20.0, 10.0, 0.0)

Line Point = (14.14, 7.07, 0.0)

now then, to figure out if the point is on the line, or outside the line:

let's say we have a line start point at: (5.0, 0.0, 0.0), and end point: (10.0, 0.0, 0.0)

this means we have a normalized direction of: (1.0, 0.0, 0.0).

we
also need to take the dot product of the direction vector and the
start/end points to check that the tested vertex is on the line(this
essentially becomes a plane equation):

(5.0*1.0 + 0.0*0.0 + 0.0*0.0) = 5

(10.0f*1.0 + 0.0*0.0 + 0.0*0.0) = 10

so, let's try 2 points: (20.0, 30.0, 0.0), and (0.0f, 30.0, 0.0).

These get plotted on the line to be: (20.0, 0.0, 0.0), and (0.0, 0.0, 0.0)

so, now we get the resulting point's dot product with the direction vector: (20*1 + 0*0 + 0*0)
= 20. we check that this value is inside the start/end point's dot
products, if it's > or < than one or the other, it means it's past
that point. as such the closest point to that line is at the end
point, and so, we return (10.0, 0.0, 0.0) as the closest point on the
line to the vertex.

(0.0, 0.0, 0.0) is 0 when you
take the dot product of the directional line, it is < 5, so that
means we take the start point as the closest point on a line to the vertex.

so the code would like:

struct Vector3{
 float x,y,z;
 
 float Dot(Vector3 &O){
  return x*O.x+y*O.y+z*O.z;
 }
 
 Vector3 Normalize(void){
  float d = sqrtf(x*x+y*y+z*z);
  if(d<=0.00001f) return Vector3(x,y,z);
  d = 1.0f/d;
  return Vector3(x*d, y*d, z*d);
 }
 //assume operator overloads for +, -, *, and constructor'sthat take's 3 floats.
};
 
Vector3 ClosestPoint(Vector3 &LineStart, Vector3 &LineEnd, Vector3 &Point){
   Vector3 Direction = (LineEnd-LineStart).Normalize();
   float LineADot = Direction.Dot(LineStart);
   float LineBDot = Direction.Dot(LineEnd);
   Vector3 LinePoint = Point*Direction;
   float LinePointDot = Direction.Dot(LinePoint);
   //check which way we need to compare LinePointDot with B and A dot products:
   if(LineBDot>LineADot){
     if(LinePointDot>LineBDot) return LineEnd;
     elseif(LinePointDot<LineADot) return LineStart;
   }else{
    if(LinePointDot<LineBDot) return LineEnd;
    elseif(LinePointDot>LineADot) return LineStart;
   }
   return LinePoint;
}
 

^^untested^^

that *should* work, but someone correct me if i just went on a completely lunatic babble that makes absolutely no sense.

edit: i should have checked out what JTippetts linked to first.

This can be done much simpler. We know that we can project a vector a onto b with (dot(a, b) / dot(b, b)) * b. We can apply this by subtracting the point by one of the points on the line, projecting onto the line vector using the above, clamping our t value if needed, and adding to that point on the line again. Some code for you:

// p0 and p1 define the line segment, pt is an arbitrary point in space
const Vector3 ClosestPointOnSegment(const Vector3& p0, const Vector3& p1, const Vector3& pt) {
    const Vector3 lineSeg = p1 - p0; // seg vec
    // saturate clamps from 0 to 1, which will get us a point between p0 and p1
    // if we want the closest point on the infinite line defined by both points we would 
    // omit the saturate
    const float t = Saturate(Dot((pt - p0), lineSeg) / Dot(lineSeg, lineSeg));  
    return p0 + lineSeg * t;
}