The projection approach isn't all that bad, all you need is the projection matrix:

P = (M

^{T}M)

^{-1}M

^{T}Like Alvaro said, use two sides of the triangle as the basis vectors, in this case we'll use (B-A) and (C-A), where 'A', 'B', and 'C' are the triangle points. Plug those vectors into the columns of matrix 'M', which makes it 3x2. Solve for the projection matrix 'P', which can be done by hand since the matrices are small, and then easily translated to code.

Now let's say you're testing point 'D'. All you do is multiply (D-A) by the previously calculated projection matrix 'P', which results in a 2D vector. This is the coordinate of the projected point in the subspace defined by the original basis vectors. Let's call the coordinate values 'x' and 'y'. It follows that that the projected point is in the triangle iff the sum (x+y) is less than or equal to 1, and both 'x' and 'y' are non-negative.

I know the explanation seems pretty intense, but once you've boiled down the math, it ends up being only a few lines of code