bool GetFacetWinding(IN IFacet * facet)
This does not make sense. There is no 'right' winding order for any triangle or polygon.
You're right - the name of the function should read as IsFacetWindingClockwise().
You could also test whether all facets of a whole object are winded consistently (for example because you know the object should be watertight).
This is exactly what I'm trying to accomplish. Point in case - the faces of a cube should be wound consistently. My test fails to show that. Note that I'm aiming to determine the winding of a 3D polygon.
In short - if I pass in a set of polygons taken from a closed mesh or polytope - even if it's concave -, I want them all to consistently return the same winding.
Also, I think (please correct me if I'm wrong) the formula you're using determine the winding only works for 2d triangles but your code also allows higher-order polygons, so your m->vertices should already be transformed into projection space or reference plane, and it should contain only triangles.
int iAxis0, iAxis1;
//project the triangle
IMath::GetMinorAxes(vNormal, iAxis0, iAxis1);
do exactly that. They use the normal to identify the minor axes (eg the two non-major axes) and the rest of the code uses the triangle that is projected onto this plane as baseline.
In fact, in your case you don't even need to use the area formula, as the sign of the z from the normal will give you the winding (assuming that the original vectors used in the normal cross-product calculation had x and y set to the pre-transformed 2d coordinates and z=0) - i.e, the normal will be pointing either towards positive z or negative z, and the x and y components of the normal will be 0.
Hm - unless I'm misunderstanding you, you're suggesting the following code should basically give me the winding?
bClockwise = vNormal[iMajorAxis] > 0;
Where iMajorAxis is the predominant axis of the vector, eg z for (0, 0, -1). I'll be honest, my brain's a little garbled at the moment but at least one case I can see where it fails is for faces that are diagonal with respect to all major planes, eg faces with (unnormalized) normals like (1, 1, 1), (-1, 1, -1) etc. That is, cases where no major axis exists (come to think of it my original code doesn't account for these either).