usually figuring out the vertex normals is done (in the simple case) via a cross product.
say, we have 3 points (A, B, C) along the edge of a polygon, and want to find the normal for point B:
Z=normalize(X % Y)
then basically, similar is done for each vertex, to generate starting normals.
note that depending on the winding of the vertices, the normals will either point into or out-of the object.
while this works, the normals generated tend to point directly out of the face, and ignore adjacent faces. for some tasks, we might want weighted normals.
the trick is basically for each vertex, finding any other vertices at the same coordinates.
then, one can do a "weighted average" of all the normals, with the weight being the dot-product between the normals.
this will smooth out soft curves while also preserving sharp edges.
dunno if this helps...
Yea, I already know about cross-product method. I have made one, but Newell's Method is more accurate.
Like I was saying, my method grabs all neighboring verts, and forms a projected face from them on each axis. For each created face, I find the normal of the face via the cross product method. Then with all those calculated face normals, I take the average and I end up with the more accurate per-vertex normal.
My problem was correctly finding the neighbor vertices in a consistent clockwise/counter-clockwise order. They often get reversed with my current algorithm. But, if I could find a way to correctly determine the order of the verts after finding them, I could simply reverse the per-vertex normal whenever it was facing the wrong direction. However I found a way to ensure the normal is facing the correct direction no matter what, and I don't need to determine the handedness of the neighbor verts order to do it. Look at explanation below.