dot(modelSpaceView, triangleDirection).

A triangle who's normal is facing away from the camera (i.e where that dot < 0) isn't necessarily back-facing; such triangles can well be visible under a perspective projection.

dot(eyePosInModelSpace-triangleVertex[0], triangleDirection);

or

dot(vec4(eyePosInModelSpace.xyz,1), trianglePlane.xyzw);

I think the arrow in the 2nd picture should be in the opposite direction, it's eyepos-vertexpos, but otherwise a good visualization of the idea

Ah yeah, if the red arrow is flipped then it does work.

I couldn't see how this worked because I was picturing it backwards... oops

Whaa..? That's not how rasterizers detect triangle orientation. See 'Oriented edges section' from Fab. Giesen. It basically exploits a mathematical property and is nearly free.

I thought GPUs just rasterized the triangle as-is and stopped the row the two endpoints crossed. This means that you get a mechanism to determine the last scanline of a visible triangle and to do backface culling all in one (since culled triangles would end right in their first row, effectively rendering nothing), and without much math in the way.

How long before this turns into a discussion of GPU internals? =P

I thought GPUs just rasterized the triangle as-is and stopped the row the two endpoints crossed. This means that you get a mechanism to determine the last scanline of a visible triangle and to do backface culling all in one (since culled triangles would end right in their first row, effectively rendering nothing), and without much math in the way.

I didn't make the link between that article which talks about points in the positive triangle area, and direction of front face and back face.

Ooops. Sorry. Right blog, wrong link. See Fill rules.

Edit: The baricentric conspiracy article is still important to understand what's going on. Through the math exposed there, the author arrives to the code:

if (w0 >= 0 && w1 >= 0 && w2 >= 0)
                renderPixel(p, w0, w1, w2); 

To determine whether a point is inside a triangle.

w0, w1 & w2 are calculated using:

int w0 = orient2d(v1, v2, p); //F12
int w1 = orient2d(v2, v0, p); //F20
int w2 = orient2d(v0, v1, p); //F01

He obtains the formula from his baricentric conspiracy math derivation, where he defines F12, F20 & F01 using a counter-clockwise scheme.

If a point is inside a triangle but the vertices are given in clockwise order, then w0, w1 & w2 will all be negative. This is a damn good mathematical property.

Counterclockwise triangles can be checked doing "if (w0 >= 0 && w1 >= 0 && w2 >= 0)"; while clockwise triangles can be checked doing "if (w0 <= 0 && w1 <= 0 && w2 <= 0)"

Another way to switch this, is to alter the definition of w0, w1, & w2 so that you use F21, F02 and F10 instead; then check for all positive for clockwise triangles, and all negative for counterclockwise ones.

In other words, checking whether a triangle is counterclock- or clock-wise is a matter of checking whether all three coefficients are positive or negative (or toggling the math formula to flip this identity). No cross product involved, no dot product involved. No need for a triangle normal at all. In fact, this can be solved using integer arithmetic and no need for floating point!

Since during rendering we only care that triangles are in one order, we don't need to do anything. We just leave "if (w0 >= 0 && w1 >= 0 && w2 >= 0)" and the vertices in the wrong winding order will automatically be left out.