I'm a little confused of smoothing groups in 3ds files. A cylinder for example must be smooth in the middle but have hard edges at the top and bottom. Does this mean that for a cylinder, 3 smoothing groups are used: one for the top, one for the bottom and one for the middle part? If yes then calculating the vertex normals for each part would mean that in order to get smooth/hard edge preservation one must apply the Mean Weighted Equally algorithm for each of the three parts separately. Is this the way to do it? Also I would want to calculate normals myself sometimes(for deformating surfaces for example). How can I create smoothing groups, or how can I go around calculating normals that don't smooth hard edges. How are normals calculated in commercial packages like 3dmax for instance, or games?

We actually discussed this recently in another thread. The basic overview...

The idea of smoothing groups is that any two adjacent faces that are in the same smoothing group (i.e. their smoothing group membership *intersects*, since each can be in multiple) should look "smooth" between them.

In practice this means that you have to do average normals at vertices, saving contributions for different smoothing "sets" separately. Then if you have more than one smoothing group set at a vertex, it may be necessary to split that vertex, assigning a unique normal to each face. There are plenty of details and it's actually pretty annoying to get it working flawlessly (even 3dsmax doesn't always work).

The easiest advice I can give you is to look at some code that I wrote quite a while ago to do it... it seems to work pretty well:

LoadCommonImpl.hpp (use username "anonymous").

Note that the code is technically open source, so you can't just rip it off, but it may be able to give you the idea.

To be honest, the best advice is just to use a tool like Milkshape to convert to a file format that supports vertex normals. 3DS is getting seriously dated...