If this is in 2D space, the math can be a bit challenging. I solved this a few years ago by digging up some musty old grade school algebra books. The general principle here is to think of your shapes as a collection of lines, or a collection of vertex points. A triangle has three vertices, a rectangle has four, a pentagon has five, etc. What we want to note is that the number of sides/vertices in a shape shouldn't really matter very much. To detect if any two shapes overlap, or intersect, we can test each line in the first shape against every line in the second shape. If no lines in the first shape intersect with lines in the second shape, then we might not have a collision (note: A shape could still be completely enclosed within another shape!).

How do we do this?

Well, we have to define a "line". Algebraically, it would be defined as "Y = mx + b", but that's a general line formula which stretches on to infinity. For the sides of a shape, you want to define a line by a starting point and an ending point. Then, you want to do a line intersection test against every other line in the other shape (We're looking at an N^2 for loop here... watch out!)

Anyways, the code I came up with is here:

/// <summary>
/// Given the end points for two line segments, this will tell you if those two line segments intersect each other.
/// </summary>
/// <param name="A1">First endpoint for Line A</param>
/// <param name="A2">Second endpoint for Line A</param>
/// <param name="B1">First endpoint for Line B</param>
/// <param name="B2">Second endpoint for line B</param>
/// <returns>true or false depending on if they intersect.</returns>
public static bool DoIntersect(Vector2 A1, Vector2 A2, Vector2 B1, Vector2 B2)
{
//NOTE: Floating point precision errors can cause bugs here. This can result in false-positives or true-negatives.
//Based off of the Y = mx + b formula for two lines.
//calculate the slopes for Line A and Line B
double mx1 = (double)(A2.Y - A1.Y) / (double)(A2.X - A1.X);
double mx2 = (double)(B2.Y - B1.Y) / (double)(B2.X - B1.X);
//calculate the y-intercepts for Line A and Line B
double b1 = (double)(-mx1 * A2.X) + A2.Y;
double b2 = (double)(-mx2 * B2.X) + B2.Y;
//calculate the point of intercept for Line A and Line B
double x = (b2 - b1) / (mx1 - mx2);
double y = mx1 * x + b1;
if (double.IsInfinity(mx1) && double.IsInfinity(mx2))
{
//we're dealing with two vertical lines. If both lines share an X value, then we just have to compare
//y values to see if they intersect.
return (A1.X == B1.X) && ((B1.Y <= A1.Y && A1.Y <= B2.Y) || (B1.Y <= A2.Y && A2.Y <= B2.Y));
}
else if (double.IsInfinity(mx1) && !double.IsInfinity(mx2))
{
//Line A is a vertical line but Line B is not.
x = A1.X;
y = mx2 * x + b2;
return (
((A1.Y <= A2.Y && LTE(A1.Y , y) && GTE(A2.Y, y)) || (A1.Y >= A2.Y && GTE(A1.Y, y) && LTE(A2.Y, y))) &&
((B1.X <= B2.X && LTE(B1.X, x) && GTE(B2.X, x)) || (B1.X >= B2.X && GTE(B1.X, x) && LTE(B2.X, x))) &&
((B1.Y <= B2.Y && LTE(B1.Y, y) && GTE(B2.Y, y)) || (B1.Y >= B2.Y && GTE(B1.Y, y) && LTE(B2.Y, y))));
}
else if (double.IsInfinity(mx2) && !double.IsInfinity(mx1))
{
//Line B is a vertical line but line A is not.
x = B1.X;
y = mx1 * x + b1;
return (
((A1.X <= A2.X && LTE(A1.X, x) && GTE(A2.X, x)) || (A1.X >= A2.X && GTE(A1.X, x) && LTE(A2.X, x))) &&
((A1.Y <= A2.Y && LTE(A1.Y, y) && GTE(A2.Y, y)) || (A1.Y >= A2.Y && GTE(A1.Y, y) && LTE(A2.Y, y))) &&
((B1.Y <= B2.Y && LTE(B1.Y, y) && GTE(B2.Y, y)) || (B1.Y >= B2.Y && GTE(B1.Y, y) && LTE(B2.Y, y))));
}
//figure out if the point of interception is between all the given points
return (
((A1.X <= A2.X && LTE(A1.X, x) && GTE(A2.X, x)) || (A1.X >= A2.X && GTE(A1.X, x) && LTE(A2.X, x))) &&
((A1.Y <= A2.Y && LTE(A1.Y, y) && GTE(A2.Y, y)) || (A1.Y >= A2.Y && GTE(A1.Y, y) && LTE(A2.Y, y))) &&
((B1.X <= B2.X && LTE(B1.X, x) && GTE(B2.X, x)) || (B1.X >= B2.X && GTE(B1.X, x) && LTE(B2.X, x))) &&
((B1.Y <= B2.Y && LTE(B1.Y, y) && GTE(B2.Y, y)) || (B1.Y >= B2.Y && GTE(B1.Y, y) && LTE(B2.Y, y))));
}
/// <summary>
/// Equal-Equal: Tells you if two doubles are equivalent even with floating point precision errors
/// </summary>
/// <param name="Val1">First double value</param>
/// <param name="Val2">Second double value</param>
/// <returns>true if they are within 0.000001 of each other, false otherwise.</returns>
public static bool EE(double Val1, double Val2)
{
return (System.Math.Abs(Val1 - Val2) < 0.000001f);
}
/// <summary>
/// Equal-Equal: Tells you if two doubles are equivalent even with floating point precision errors
/// </summary>
/// <param name="Val1">First double value</param>
/// <param name="Val2">Second double value</param>
/// <param name="Epsilon">The delta value the two doubles need to be within to be considered equal</param>
/// <returns>true if they are within the epsilon value of each other, false otherwise.</returns>
public static bool EE(double Val1, double Val2, double Epsilon)
{
return (System.Math.Abs(Val1 - Val2) < Epsilon);
}
/// <summary>
/// Less Than or Equal: Tells you if the left value is less than or equal to the right value
/// with floating point precision error taken into account.
/// </summary>
/// <param name="leftVal">The value on the left side of comparison operator</param>
/// <param name="rightVal">The value on the right side of comparison operator</param>
/// <returns>True if the left value and right value are within 0.000001 of each other, or if leftVal is less than rightVal</returns>
public static bool LTE(double leftVal, double rightVal)
{
return (EE(leftVal, rightVal) || leftVal < rightVal);
}
/// <summary>
/// Greather Than or Equal: Tells you if the left value is greater than or equal to the right value
/// with floating point precision error taken into account.
/// </summary>
/// <param name="leftVal">The value on the left side of comparison operator</param>
/// <param name="rightVal">The value on the right side of comparison operator</param>
/// <returns>True if the left value and right value are within 0.000001 of each other, or if leftVal is greater than rightVal</returns>
public static bool GTE(double leftVal, double rightVal)
{
return (EE(leftVal, rightVal) || leftVal > rightVal);
}

It could probably be improved by someone smarter than me, but it works well enough for the time being. Note that math like this is susceptible to *floating point precision errors *so I had to write my own floating point comparison "==" methods with a small epsilon value. I'm still not 100% confident this is perfectly bug free.

Also, since the method I suggest above uses an N^2 approach, you might want to consider using a *broad-phase* and *narrow-phase* collision check (if you notice performance problems!). The broad phase check could just be sphere vs sphere intersections, where each sphere completely encloses a shape. It's very fast to implement and doesn't cost much. If the spheres intersect, then you can run the N^2 check for more precision.

There's one other thing you might want to consider: If you're just trying to use one shape to overlap another shape, why don't you just toggle the drawing order of the shapes? Is there a game logic reason behind it?

**Edited by slayemin, 16 April 2014 - 12:01 AM.**