Quote:
I think if every polygon is on the front side of the plane of every other polygon, the group is convex.
equals this:
Quote:
Each polygon in the set is entirely contained in the half-space in front of every other polygon in the set.
Initially you didn't think this was the answer you were looking for, but you may not have understood what I meant.
Any planar polygon divides 3d space into two half-spaces, the space behind the plane of the polygon, and the space in front of the plane of the polygon.
A polygon can be tested against the plane of another polygon as follows. If all the verts of polygon A are on or in front of the plane of polygon B, polygon A is front of polygon B. If all the verts of polygon A are on or behind the plane of polygon B, polygon A is behind polygon B.
In a convex set of polygons, each polygon will be in front of the plane of every other polygon in the set (make sure to include an epsilon in your plane test). So a function might look something like:
bool IsConvex(Poly polys[], int numpolys)
{
for (int i = 0; i < numpolys; i++)
{
for (int j = 0; j < numpolys; j++)
{
if (i != j && polys.Test(polys[j]) == BEHIND)
return false;
}
}
return true;
}
That's just off the top of my head, so I can't swear by it. But I think something like this is the answer you're looking for.
[Edit]: sorry about the formatting.