Could you please give me the detailed calculations of the normal of a surface formed by vertices below? I calculated them myself but when I use them in my OpenGL program they give me wrong results. I just want to compare my calculations and make sure that I’m doing them right. P0 = (-1.0, 0.0, 1.0) P1 = ( 0.0, 1.0, 0.0) P2 = (-1.0, 0.0, -1.0)

That’s the result I got but according to a sample app the normal is:

N = [-2, 2, 0], normalized N = [-0.707, 0.707, 0.0]

Which doesn't make sense and it’s driving me insane!

This is a typical conflict of 3d coordinates' system handedness. Imagine aligning your right thumb with the direction from P0 to P1, and your right index finger with the direction from P0 to P2. A right handed system would interpret their cross product, as pointing to the direction of your right middle finger (when you stretch it out perpendicular to your palm). A left handed system would yield the exact opposite direction.

As JohnBolton said, this has immediate impact on the 3d api's ability to determine the visible side of a triangle. If you port geometry created in a left-handed system, into a right-handed one, (or vice-versa) you'll have to switch the declaration of vertices in all faces, as such:
1. change vertex order from {v0,v1,v2} into {v0,v2,v1}
2. Multiply your view matrix with the reflection matrix

    [ 1  0  0  0 ]R = [ 0  1  0  0 ]    [ 0  0 -1  0 ]    [ 0  0  0  1 ]

Let V be the view matrix. If you use column-major format use as a view matrix the product V*R, else R*V.