You could go for an old trick where the alien races have limited resources, and Earth is a figurative gold mine. It could begin from attempting to fight off the first invasion, which nearly wipes out the entire human race. Then The Empire is formed with the remaining population, and you reverse engineer the aliens and what they had to gain a basic advancement, sort of leveling the playing field without making you strong yet.
Additionally, in relation to focusing on one Empire, you could say that due to their limited resources much dissension within the alien races was created, and so their is no significant power left. The problem you arise with there being only one Empire, is that there is no plausible reason (that I can think of) for any significant advancement in power/numbers over time.
On the note of the 'Eliminators', I think it could work quite well. Perhaps the reason Earth was attacked now is because a prophecy, or event, has taken place that revealed to them their limited time before the 'invasion'. Over time, through defending the Empire you created and collecting information on your enemies, you discover this. By that time, you have become a lot more powerful, and so you will shift away from the minuscule fights with the aliens and face the much larger threat at hand. This could easily explain a change in difficulty as well.