I guess you should post the math for testing the intersection so someone can suggest a better algorithm or add something to it to make it work properly.

I did something very similar in my ray tracer for ray-triangle intersection. For what it's worth, I got the ray-plane intersection from Wikipedia and the triangle intersection from SoftSurfer. While I used pretty much the same method that Grumple posted, I'll post my code just to give another perspective/implementation that will hopefully help it be a little more clear:

struct Triangle
{
Vector3f a, b, c;
Vector3f normal; // normalized


bool intersects(const Ray& ray, float& t, Vector3f& n) const
{
// from http://softsurfer.com/Archive/algorithm_0105/algorithm_0105.htm (and Plane.intersect())
n = normal;

float num = normal.dot(a - ray.start);
float den = normal.dot(ray.direction);

if (num != 0 && den == 0)
{
return false; // parallel, but never intersect
}
else if (num == 0 && den == 0)
{
t = 0; // parallel and in same plane
return true;
}
else
{
t = num / den;
}

if (t < 0)
{
return false; // intersection point is behind the ray's start
}

Vector3f u = b - a;
Vector3f v = c - a;
Vector3f w = ray.start + ray.direction * t - a;

float uv = u.dot(v);
float wv = w.dot(v);
float vv = v.squaredLength();
float wu = w.dot(u);
float uu = u.squaredLength();

float s1 = (uv * wv - vv * wu) / (uv * uv - uu * vv);
float t1 = (uv * wu - uu * wv) / (uv * uv - uu * vv);

return s1 >= 0 && t1 >= 0 && (s1 + t1) <= 1;
}
};

It's important to be aware that this degenerates when the ray lies in the triangle's plane, because I simply set [font=courier new,courier,monospace]t = 0[/font], when really [font=courier new,courier,monospace]t = 0[/font] could be outside the triangle. You could optionally return false if [font=courier new,courier,monospace]t = 0[/font] and since a triangle is infinitely thin as well. It just depends on how you want to handle this situation.

With really small errors like that, it could very easily be floating point rounding errors. Try replacing the intersection code with double instead of float and see if the error is smaller (this is just to make sure it is indeed floating point accuracy errors).

I'm going to go ahead and assume it is indeed floating point rounding issues, and to be honest there isn't a fool-proof way around this. One commonly used solution would be to use some epsilon to allow for a controllable degree of error (which is similar to what you mentioned). For example:

float epsilon = 0.001f;
return (s1 >= 0.0f - epsilon) && (t1 >= 0.0f - epsilon) && ((s1 + t1) <= 1.0f + epsilon);


Another thing would be to look at your compiler's optimization settings. There are usually some flags that can be set in the compiler that will trade accuracy for speed (usually I've found it to be accuracy is favored over speed by default, and you have to explicitly say to favor speed over accuracy). Double check them and make sure you aren't sacrificing accuracy for speed.

I've never had precision errors this small cause any problems for me though, so I can't give any more solutions. It also depends on your data. If you're mixing large numbers and small numbers in calculations, the errors will be much more noticeable. Try to make sure your floats don't differ by too many orders of magnitude if you want more precision.