(EDIT: Just to be clear, I mean that given a triangle A, the vertices A1, A2 and A3 shouldn't all have identical normals - rather, their normals should be influenced by the normals of other triangles as well, such that, for example, normals A2 and A3 should have a normal also influenced by the normal of the separate triangle A2-A3-A4.)
So, first question - is that indeed the way to achieve smoother-looking shading?
If it is, I've got something of a problem, as the image below (which should be a pyramid) no doubt makes clear. The triangles themselves are all placed accurately, but I'm not calculating my normals properly. I should mention that the way I build my triangles internally is that I have a fixed set of vertices and, based on a number of parameters, choose some of these vertices as points in an arbitrary mesh (which are first stored as pointers, and then ultimately uploaded to a vertex buffer object as a series of floats). Since vertices store their normals as well as their position, I thought it would be easy to ensure that each time a vertex is used by a triangle, that triangle's normal influences the vertex's total normal. It appears that isn't the case.
Currently, every time I make a new triangle, I calculate its normal and then add that normal to the existing normals of the three vertices involved. After making all my triangles, I reduce the length of each vertex's normal to 1. I've also tried adding them and then dividing by the number of triangles involved, but that doesn't work and it doesn't sound like it should work, either.
This is my normal calculation. I run this for all triangles (which contain pointers to their Verts 1, 2, and 3).
float A[3] = {Vert2.x - Vert1.x, Vert2.y - Vert1.y, Vert2.z - Vert1.z};
float B[3] = {Vert3.x - Vert1.x, Vert3.y - Vert1.y, Vert3.z - Vert1.z};
float nX = A[1] * B[2] - A[2] * B[1];
float nY = A[2] * B[0] - A[0] * B[2];
float nZ = A[0] * B[1] - A[1] * B[0];
nX *= -1;
nY *= -1;
nZ *= -1;
Vert1.nx += nX;
Vert1.ny += nY;
Vert1.nz += nZ;
Vert2.nx += nX;
Vert2.ny += nY;
Vert2.nz += nZ;
Vert3.nx += nX;
Vert3.ny += nY;
Vert3.nz += nZ;
This is how I normalize the normals. I run this for all Verts after having run the above code for all triangles.
//Normalize!
float Length = sqrt(Vert.nx * Vert.nx + Vert.ny * Vert.ny + Vert.nz * Vert.nz);
Vert.nx /= Length;
Vert.ny /= Length;
Vert.nz /= Length;
Clearly, the results aren't what I want. I was hoping somebody might be able to help me arrive at a better solution than this. Any advice?