arbin,

This is a well studied problem. To properly calculate the normals you must have a consistent facet winding (or know the winding of each facet -- which is equivalent) and know the proper orientation (outside vs. inside) of at least one facet. In this case you can then propagate that normal orientation to the rest of the surface. I don't know where you heard the rumor about the cross product, but it is untrue. If your faces are wound consistently, the cross product can consistently yield the proper normal direction. So, long story short, with an arbitrary mesh and no known orientation, this is an ill posed problem and cannot be solved. However, the case in real life is almost never this bad as almost all of the models with which you will have to deal should have a consistent winding, and you just have to know which one that is (right-handed vs. left-handed). If your error is just in rendering, another simple approach you might consider is just to use 2-sided lighting. Each vertex gets 2 normals, calculated with opposite windings. Then the lighting calculation is carried out for both sides of the surface and everything is lit "properly".

Well, mathematically the normal of a plane would be calculated using the cross product of two non-parallel vectors in that plane and since a triangle lies in a plane, you'd calculate the triangle's normal by taking the cross product of two edges. Since the length of the resulting normal depends on the angle and length of both vectors you'd normalize the result for proper lighting.

This wouldn't work well for degenerate triangles, but the graphics card wouldn't like them either.

In order to get smooth normals, i.e. an average of the normals of all triangles sharing a point, there are several methods that can be used.

The following has worked for me:

sumNormals = (0,0,0)sumAngles = 0for each triangle sharing a point  sumNormals += normalized normal of triangle * angle at that point  sumAngles += angle at that point//since all normals added are normalized, this should yield a unit-length normalnormal = sumNormals / sumAngles

This way, the contribution of each triangle to the smoothed normal is weighted and you should get reasonable results.

For situations where large and small triangles share a point, you might also take the triangles' area into account.

If you don't know if the list of vertices that make up a triangle are given in a clockwise or counter clockwise ordering, then no, you cannot reliably compute the normal. This is because the cross product is antisymmetric; that is cross(v1, v2) = -cross(v2, v1). Thus, if your triangles are oriented inconsistently, the outward direction for some triangles will be the inward direction for others and vice-versa. A consistent orientation is a prerequisite for computing a consistent set of normals for the mesh. In a worst case scenario, you can always orient your mesh by hand using a tool like meshlab and then load it into your program with a consistent orientation.

Additionally, if your vertices are not wound consistently, you'll get problems with backface culling, which you should enable for performance reasons (less fill rate, fragment shader execution etc.).