You do not have to do any iterations. In the case you described, you do not need a force preventing penetration, because the point will never penetrate the plane. If it was a polytope with only the one extremal point touching the plane, the center of mass would accelerate towards the plane, but the one point would stay on the plane because of the rotation. Your algorithm should then detect when the 'next' point would penetrate the plane and adjust the momenta accordingly. This would have the side effect of slowing down the rotation.

edit: concerning the algorithms I cannot really help you. Implement something you find easy to understand i guess.

You do not have to do any iterations. In the case you described, you do not need a force preventing penetration, because the point will never penetrate the plane. If it was a polytope with only the one extremal point touching the plane, the center of mass would accelerate towards the plane, but the one point would stay on the plane because of the rotation. Your algorithm should then detect when the 'next' point would penetrate the plane and adjust the momenta accordingly. This would have the side effect of slowing down the rotation.