Consider that the box is formed by the points A,B,C,D

A B
C D

From these points, you can obtain the equations of the lines forming the edges of the box: A->B, C->D, A->C, B->D. Recall that the parametric form of a line can be expressed as:

P=P1+u*(P2-P1)

for any 2 points, P1 and P2, on a line. This means that any point along the line can be expressed in terms of the parameter, u. A value of u=0 evaluates to P1 on the line, a value of u=1 evaluates to P2 on the line, and values of u in between 0 and 1 evaluate to points on the linesegment described by P1 and P2. Values of u less than 0 or greater than 1 evaluate to points beyond the ends of the line segment.

For a given point, P, you can find the value of u that corresponds to the closest point on the line, P1->P2, by:


u=((P.x-P1.x)*(P2.x-P1.x) + (P.y-P1.y)*(P2.y-P1.y)) / length(P2-P1)^2


In essence, you project the point, P, onto the line. You calculate the dotproduct of the vectors P1->P and P1->P2, and divide this dot product by the length of P1->P2 squared.

This will obtain the value of u for that particular line that denotes the point on the line nearest to P. Since you are testing for distance to a box made of line segments, then you have to account for the fact that this nearest point might not lie on the box. In this case, you can simple clamp the value of u to the range [0,1]. If u is less than zero, then the closest point on that line segment to P is the point P1. If us is greater than 1, then the closest point is P2.

Once u is clamped, you can plug it into the parametric form of the line being tested, to find out the actual coordinates of the point. Perform this test for all four of the line segments forming the box, then calculate the distances between P and the closest points, returning the smallest of these four distances as the result.

Some sample Lua code is as follows:


function closestPointOnLineU(p, p1, p2)
local length=math.sqrt((p2.x-p1.x)*(p2.x-p1.x) + (p2.y-p1.y)*(p2.y-p1.y))
return ((p.x-p1.x)*(p2.x-p1.x) + (p.y-p1.y)*(p2.y-p1.y)) / (length*length)
end

function closestPointOnLineSegment(p, p1, p2)
local u=closestPointOnLineU(p,p1,p2)
u=math.max(0, math.min(1, u))
return {x=p1.x+u*(p2.x-p1.x), y=p1.y+u*(p2.y-p1.y)}
end

function distanceToLineSegment(p, p1, p2)
local closest=closestPointOnLineSegment(p,p1,p2)
return math.sqrt((p.x-closest.x)*(p.x-closest.x) + (p.y-closest.y)*(p.y-closest.y))
end

function distanceToBox(p, A, B, C, D)
local d1=distanceToLineSegment(p, A, B)
local d2=distanceToLineSegment(p, C, D)
local d3=distanceToLineSegment(p, A, C)
local d4=distanceToLineSegment(p, B, D)
return math.min(d1,d2,d3,d4)
end

Note that it's quick code, so I might have made an error somewhere, but initial tests indicate it works.

Heh, yeah, alvaro's code is the way to go if your boxes are axis-aligned. The algorithm I presented will work for any arbitrary box, but is overkill if your boxes never rotate. And looking at his, I realized that I didn't include the tests for if a point is inside the box, return 0 distance. As it stands, the code I presented will give the distance to the box's wireframe, so points inside the box will have a non-0 distance value. On an arbitrary box (as long as it is a box, and not a rhombus or trapezoid or other funny shape) you can test this easily by checking the values for u of the A->B and A->C tests, and if both do not require clamping (ie, both are in the range of [0,1]) then the point is inside the box, and you can return 0 for the distance.