Unless of course you have a more advanced physics system where objects can change size.Also, I would say that in order for a collision to take place, one of the colliding objects must have moved. No need to test 2 objects together if neither of them moved.
But yeah, identifying objects that have not changed in any way and skipping those pairs, is a very useful optimisation to use at some stage in the collision detection process. It likely doesn't matter too much at what point you do it.