You can make it such that some weapons are more or less accurate at greater distances, or deal more/less dmg based on distance. Players would have to mix/match ships to make sure that whatever range ships they come across can't find a blind spot. Also, you could have some ballistic weapons (Such as missles/mines/etc) and Energy weapons. Ballistics are good vs. Shields while Energy is good vs. Shields. Then give every ship a Hull/Shield health based on size/expense. So small fighters have low hull but high shields, Battle Cruisers with High hull but low shields and Dreadnoughts with High Shield, High hull but loss of something else (Mobility most likely) b/c the generators are spending all the power powering the shields. Players would have to mix match ships to ensure they could destroy both types of defense. so you have 100 battle ships, range=50-150 vs. 50 battleships optimal range=50-150 50 fighters optimal range =25-75 The battleships would have start getting picked off by the fighters if they managed to get closer than the battle ships min range. So, as long as a player can not obtain a ship that 1) Has a range that encompasses the enemies ranges 2) Has weapons that can take out both shields and hull they will need to mix/match fleets or worry about coming across a fleet that exploits their weakness. Beyond that you could create utility ships... ships that help the targeting systems on nearby ships, or ships that repair nearby ships, or cloak... so on so forth.

About RPS (rock-paper-scissors) mechanic:

(assuming the "right" ship deals +100% damage to the "right target" ship)

Let's say one fleet is 30 ROCK ships. The second fleet is 10 ROCK, 10 PAPER, 10 SCISSORS ships.

Would the second fleet end up the winner here? (note: targetting rules (what ship targets which in what order) can be completely arbitrary if you wish)

This is a tough one, I have seen some games where it is often in your best interest to Rock, and Rock as much as you can, and then anyone who built a balanced force or didn't go all Paper will lose. (Hello Warhammer 40k) Preferably, a well played balanced force will win out over a very unbalanced force.

Depending on how Rock beats Scissors, or Paper beats Rock, I could see the fight going different ways. Perhaps Rock are very tough, slow ships, while paper are very long range ships, while scissors are very fast ships. In this case, the Rocks would crush the Scissors, smash through the other Rocks (not without damage), and then lose in a battle of attrition versus the Paper ships. Another example might be Rock being swarms/drones, Scissors being great single target smashers and Paper being weaker AoE attacks.

It's a fine balance determining how much having more numbers makes a difference. Especially in a space game where everything is ranged combat, it can end up being too easy to have a larger and larger force crush a smaller force with less and less attrition. Which tends to make a force that is all one type more likely to be more prevalent and exploitative.