For posterity, I found the problem. It was in the frustum setup. I have set the near plane to 0.001f and far to 10000.0f which caused some rounding errors probably. Changing the near plane to 10.0f eliminated the problem.
Looks like z-fighting. The curling part of the page (on the cylinder, as you say) looks like it's actually partially intersecting the flat part of the page - make sure your depth buffer is properly setup.
Thanks for your reply!
If I understand it properly z-fighting takes place whenever the z values of the vertices are very close, but in my case the weird thing is that they are very well separated. Please explain me this if I'm wrong.
Thanks for your really good help and answer. Now the balls dont overlapp each other and this problem is gone. But the method you gave me for collision response acts like it didn't conserve momentum. And that's not what I'm trying to achieve. After some time the balls just slow down, and it looks like kinetic energy would get lost somewhere. Is it because of the coefficient of restitution? If i set it to zero then the balls don't respond to the collision. But the bigger it is the faster the balls are loosing on speed. Oh and in addition i was rather trying to make the answer being mass dependent. Once again thank you, coz you were a great help for me! If you would just give me an answer to the second question then i would be grateful.