I tried moving the removing objects into the collision statement but that doesn't work. I'm stumped.

It checks collision twice because it first checks collisions on A against all B, and then collisions on B against all A, right? Maybe set a flag on your objects that specifies whether they've been checked already, and skip them as needed?

EDIT: No... That's not it... I misread your code.