I have this problem, there are various races (factions) and they can move ships over systems to fight. But it gives me a headache a situation when more than 2 parties are involved.

I don't want a complex multi sides battle system, I really don't want it. The typical situation is 1 vs 1 battle and that's what will be happening most of the time. But sometimes 2 conquerors might move on the same defended planet and... yeah, what then? Who attacks whom?

Note: that battles can take more than one turn so a trick where we ressolve a battle fully after each race's move won't work.

Note 2: the game is assymetrical, 1 player and X computer opponents and their rules don't need to be the same (battle rules for computer could be different - actually in some cases these will be different for immersion purposes), so we are not limited by traditional turn order or anything like that

Well, one solution might be to avoid the problem: claim that the large accumulation of faster-than-light drives, shields and energy weapons prevents further ships from jumping in via whatever FTL mechanism you have; whoever arrives second to the battlefield is left in hyperspace/Alcubierre bubble/whatever until the battle is resolved and the attendant interference has subsided, at which point they pop into space and give battle themselves.

Another, similar--but more drastic--idea might be to decide that the accumulation of faster-than-light drives, shields and energy weapons outright destroys any further ships that jump in: the last party to the battle loses by default. (Which might have some interesting effects. For example, the player orders an attack on planet X, with a fleet due to arrive in three turns. Ubeknownst to them, an AI has already ordered an attack on X; two turns later, the AI fleet shows up and battle is joined. The player now has to hastily call back their ships--presuming that they notice the battle and don't lose the fleet to inattention--losing the initiative and leaving a portion of their forces unavailble in hyperspace/Alcubierre bubble/whatever while they return to base.)

With the given scenario (two attackers independently attacking the same planet) I would have the attackers fight each other, then the survivor fights the planet. This gives an advantage to whoever owns the planet itself, and mirrors a bit of history - occasionally two independent armies WOULD go for the same fortified place at the same time, and when that happened they would often tear each other to pieces over the "right" to take the planet, weakening themselves past the point of actually being able to take it.

So for any fortified or otherwise conquerable area, that's how I'd do it. If the fight lasts more than one turn, the fortified area fights no one - unless they choose to specifically attack one fleet, maybe?

For three different fleets meeting out in space somewhere (assuming that is possible in your game), you can try to decide on other factors (do fleets have a "speed" attribute of any kind? If so, the two fastest fight first), or have the game try to guess what will give the player the most fun - and what THAT means is entirely up to you. As battles can last more than one turn, you may need to have a system where either defeated fleets "flee" a short distance or multiple hostile fleets can actually occupy the same tile, even if only two can fight each other at a time.

what happens if two lions seen a gazelle? They have only two options: fight each other (probably, and good for the gazelle) or form an alliance.

Well, one solution might be to avoid the problem: claim that the large accumulation of faster-than-light drives, shields and energy weapons prevents further ships from jumping in via whatever FTL mechanism you have; whoever arrives second to the battlefield is left in hyperspace/Alcubierre bubble/whatever until the battle is resolved and the attendant interference has subsided, at which point they pop into space and give battle themselves.

But I like the idea of "arrived first - something", or another arbitrary rule. Maybe give each race an initiative value and they fight based on this?

Another, similar--but more drastic--idea might be to decide that the accumulation of faster-than-light drives, shields and energy weapons outright destroys any further ships that jump in: the last party to the battle loses by default.

your heuristics should assume that the player always fights in the first wave. This creates consistency for the player, and makes them understand that, ultimately, THEY are the bigger threat. It also minimizes the amount of enemies fighting one another when facing off these types of scenarios (which might result in an undeserved win for the player that fights off against scraps).

the fortified area fights no one - unless they choose to specifically attack one fleet, maybe?

Note: how about sightly different rules for space combat and ground combat? Like the planet attacks all fleets orbiting it, but on the ground it only defends passively?

Player vs AI 1 (strongest)
Player vs AI 2 (weakest)
AI 1 vs AI 2

I think I will also use a trick in the AI code. The compurer race will have a 50% chance to resign from the moving to a target planet if there is a battle going on there already (which also is logical and benefitial to that race most of the time). Should reduce the number of occurences like multiple battles.

But I also have another question. Let's say there is a ground combat in progress. Race A invaded the planet. Now, race B send transports with troops to that planet as well. Should race A try to shoot down the incoming transporters or let them land?