Well, if you mainly invade other races it has a feel of a young empire. Agressive, flexible, young. It could be fun I guess, but I feel a space empire is better portrayed as an old one. I mean an old, stagnant empire rotten to the core with bureucracy, falling apart, riots on the streets, usurpers to the throne, court politics, internal intrigues and assassinations. And on top of it some young alien race attacks and you don't have the time to deal with it properly because you have "more important" internal affairs to take care of

I think such scenario has better impact and lets you feel how it is to be a space emperor.
I still wonder how important the tower defence part should be. Maybe even I should remove/minimize it and go for regular "disaster events"? Like every 50 turns there is a major disaster you need to deal with (civil war, rise of an usurper, alien invasion, military coup attempt, the local sun turning into supernova) and a minor one every 10 turns (like assassination attempt, rampart corruption among officials, riots on one planet, pirates). These could be done as some sort of "random cards", not as a separate minigame. You get the event and you have a list of resurces (with alternatives) you need to apply to stop the event, or a list of mini goals to achieve to stop/reduce it. Like for riots you could send a regiment of an imperial army (if you have one) or some charismatic leader to solve it peacefully.
It's a cool idea but I think it's going to be hard to both build an empire and have it constantly under siege like a tower defense game and make it not really irritating to a player. Building outer colonies only to have them steamrolled sounds not fun, and I think players will have the tendency to turtle up, emphasizing the tower defense part over the empire building part... which isn't what you are going for.
Yes, there is such danger... But I don't meant a pure tower defence game, it can be done as a sheduled attack, not even most of the time. I meant something more along the lines of this:
http://silverlemur.com/minigames/wiztowersim.phpAnyway, the concern of it being too important and distracting from the empire building part is valid...
Plus if you are constantly fighting over frontier worlds, how does the game actually end?
Well, for this kind of game I could even go for "the game ends after X turn" model. Not the prettiest one, but assuming it ends up being a short game it would be acceptable. Or I could go for victory points system, you need to get 5 victory points, you get 1 such point for doing a big thing (like constructing transdimensional gate, defeating alien race, building death star capable of destroying whole planets, etc).
An example how such game could be done (event cards driven empire building with few planets):You have 8 planets (you start with one, but can annect the others quickly). Most/half of these planets would be inhabited from the start and you would be annecting these (military or diplomacy), not colonizing. The rest would be colonized when you feel like it. Each planet would have on average 2 races (multirace empire) that not always love each other (internal problems). There is a dozen of empire wide stats (integrity, stabilization, corruption, happiness), if these fall too low (or go too high in case of negative stats) it means trouble (or even instant game over). Regularly there are disaster events, you better deal with these of the empire stats will fall down. You have/build some assets (imperial army, fleet, officials, special imperial inspectors, advisors, governors, security forces, institutions, organisations), you use these to deal with the disasters. You can improve planets (buildings, garrison forces, local institutions). You can improve empire as a whole (imperial senate decrees, projects, policies, laws). You can send expeditions to explore "outer space", it's a kind of cheap replacement of exploration part, you just decide on funding and tell them which "sector" of the galaxy they should survey and then the survey team bring you alien artifacts, contacts with other races which allows trade routes and other random goodies (or space viruses that start an empire wide epidemy

).
Note that this all sounds very rich, complex and like completely separate features, so the only way to implement it reasonably is via some abstract general mechanic (like in most card games where you have a card telling you that you encountered pirates and need 5 points of military to deal with this card and a card that tells you there is an assassination attempt and you need 3 points of security to deal with it; while these tells completely different story, it's still the same (or very similar) mechanic used).