Are you try to avoid storing or recalculating all the triangle normals?
I'm not 100% sure about that, to be honest. The reason I was going for this approach is that the models can get quite large and complex, and I figured temporarily storing triangles as pointers to vertices, rather than as sets of new vertices, would reduce the amount of data being moved around in each model, as well as facilitate sharing of vertex normals between triangles. In the end I do store triangles as a series of floats, though, since the vertices are generated and discarded as needed, and since uploading floats to the GPU as VBOs seemed more straightforward.
Honestly, though, I'm new at programing meshes like this and I'm not sure what the best approaches are.
Another thing: you should weigh your triangle normals in the vertex normal calculation. Not all triangle normals affect the vertex normal in the same amount. Consider a cube. If you want to smooth shade the cube (I know it's stupid) you know that the vertex normals should point in a 1,1,1 direction for example (I didn't normalize it, but you get the idea). A vertex in a cube is shared by 3 sides, but that means four triangles in some cases. That means a side is represented by 2 triangles at some corners. Without weighing, it will distort the desired result because you add that normal twice.
Oh gods, that does makes sense. That's surely at least a part of the problem. Hm... Could you elaborate a little on your angle-weighting? Do you mean the angle of the line defined by two vertices (or vertex locations) shared by two different triangles?
That said, your pyramid will look distorted anyway, because you smooth sharp edges too. Take a loot at a smooth shaded triangle in any 3d modelling software. It will look something like that.
Yes, I was expecting it to look weird since, as you said, it's got sharp edges. However, I was rather expecting the weirdness to be uniform along the edges of the pyramid (like the edges were highlighted or something), rather than jagged. Presumably, if the weirdness was uniform, it could be reduced by having a smoother model.
In some of the more complex models I have, there are sharp edges (90° or less) that I'd rather not have smoothed, but duller edges that I would like to smooth out. I read a little about smoothing groups on Wikipedia just now, but I'd need some kind of algorithm that can efficiently calculate and modify smoothing groups on the fly, because my meshes are supposed to be not only arbitrary but also, to an extent, deformable. Ideally, an algorithm that identifies whether a neighboring triangle is at too sharp an angle to be included in the normal modification process, or something similar. I suppose I could just compare triangle normals and, if they are too different, ignore the neighboring triangle in the calculations...