I think, going into direction of "even more smart" is pointless... Even if I manage to fix it and make it work, is it really fun? This Spock like strategy
Maybe go somewhere into realm of emotions? AI being afraid, hating someone, wanting to take revenge...
As a quick fix I guess I will make some sort of "tired" counter, it increases each turn you fight over that planet (and decreases slowly over time). If you are too tired fighting over a certain planet (reach certain thereshold) it gets desirablity halved/nullified. With exceptions, like you never will get tired defending your homeworld
Smart & cowardly races would have that thereshold lower (quicker to back off), which adds personality to aliens I guess.
An alternative approach, which depends strongly on game rules (fleet movement has to take enough turns): send a reasonably sized fleet to the very good planet, but reroute it to another nearby objective (or to retreat in extreme cases) if the good planet is too well defended, or expected to be when the fleet arrives. This way every fleet fights against appropriately small defenses, or retreats with little harm, instead of suffering unusual losses.
In case two large invasion forces meet, instead of mutually annihilating for the benefit of all other factions they would stop, conquer what they can from neutral parties and other sides, and gradually send away excess ships from places they don't want to fight at.