I am interested to know how far physics engines (commerical or otherwise) go to preserve momentum after collisions. I wrote a very simple physics system that detects when objects penetrate each other (after update via ODE solver for a small time step) and simply reset the states to that of before the update, closest points between the object are then assumed to be the collision points and used to compute collision response. Clearly, total momentum before and after collision will not be the same (assume friction coefficients of 0) and the difference can be reduced (but not completely eliminated) by trying to update again using half the timestep. So how do other physics engines handle this?

I don't have an absolute answer for you, because I don't know the internals of all the engines. For engines that use penalty-type methods, there is pretty much guaranteed to be a loss or gain in total momentum of the system. (Hopefully not a gain in energy.) But, for engines that use closed-form collision response via an impulse-momentum approach, then total momentum would be conserved. (Engines often don't implement this approach, since it becomes difficult to manage when there a body can experience multiple simultaneous collisions, or when collisions occur at different times within a time step.)

If you need a good physics engine, check out newton, which does preserve momentum and energy very good, enough good that it can handle a Newton's Cradle very well, without task-specific cheating or hacking.

See demo here:
http://physicsengine.com/downloads.html