- tanks: take a lot of dmg before going down. Have an ability to taunt enemy to attack him (eg. this could be an active taunt, like a tractor beam or a passive taunt like a low damage area effect over time).

I use the same approach. Currently my game has ships that act as "mean forwards" because they have lasting power, but their firepower is relatively low.

- damage dealer: lot of damage output, but long reload time and quite weak hull integrity

and my backliners act as glasscannons (easy to kill, but fire very powerful missiles from a distance).

- shield: ships which can project a shield on other ships

Since many of my ship designs have shields, this wouldn't apply, but I use the concept of projected shields which meshes well with the idea of "command ships".

You put one "command ship" that has a lot of shields to protect smaller vessels, and you give it an aura ability which strengthens fighters or escorts, and this could very well be more cost-efficient than just battleships. However, your command ships is more vulnerable (because he is alone for example).

- repair: ships which can repair other ships

That theoretically makes sense, but in practice, sacrificing a ships slot to repair another may be impractical and hard to balance, unless it is a) very cheap or b) has some firepower of its own, or sufficient defenses to warrant being used.

- ammo/energy: ships which provides energy beams/ammo to quicken the reload times of other ships

I buy into the ammo idea. Most ships should "fly" with bare minimum stocks to save on fuel, and as such, should have only limited cargo. In sustained combats that lasts a few "days", these ships could come in between salvos to restock your ships. Assuming your army is made of "tanks", you could survive battleships for a few encounters until they are without ammunition and at your mercy. In this specific case: tanks+ logistics would win over battleships.

In my game, I tend to keep cargo capabilities fairly low on small ships except on frigates. My ship classification insists that a frigate is generally lightly armed, but has decent armor and stellar cargo capacity. They are, essentially, armed freighters. This meshes well with the above idea.

More importantly, frigates don't need to be part of the fleet to be relevant. They may not even need to be part of your combined arms tactics to still be used. They can just ferry cargo from your planets to the location of the expected engagement (back and forth in loop). They are even hard to intercept by small fleets because they can still pack a punch.

- short/long range attacks: most effective at a certain range

Most definitely. In my game, most battleships triumph at medium to low ranges which leaves long range open for dedicated crafts.

More importantly:

I use maneuverability as an important metric: basically, how fast a ship can move, and how sharp it can turn allow me to determine how likely it is to be hit by the enemy. As a result, even at close ranges, the Battleships don't necessarily make short work of "anything". A dedicated squadron of light and maneuverable gunships, for example, could dodge most bullets with ease, taking several losses, but successfully sinking a battleship (making it an economic win overall).

In these situations, an opponent fielding just battleships is asking for trouble: it is actually a golden opportunity.

Battleship divisions should always be accompanied by smaller crafts that either focus on rapid-fire proximity shots (point-defense) or have sufficient speed and maneuver of their on to chase fast crafts and defeat them.

Another tactic I use is to have relatively the same amount of weapons on each ship (battleships might get 15-20 whereas a gunship can have 4-10). That way, it emphasizes the ability to outnumber the enemy. When all else fails (and you are stuck facing superior enemy ships such as battleships) you can just regroup all of your ships and defeat them through sheer advantage on weapon count. Since battleships will have fewer weapons that deal more damage on average, they will take down ships more easily, but a significant portion of the damage will be lost due to overkill (the last shot on every ship will have a higher % of the damage "lost"). Having more ships with lower damage results will allow to minimize the overkill ratio, and thus, maximize actual damage dealt.

So: Range, maneuver, number of weapons, role classification (tank vs glasscannon vs support ship) should give you a large assortment of things to play with. Everything else is down to execution.

Please avoid using MMO tropes, I do hate them, they always feel super game-y.

A simple Rock-paper-scissors approach might work

weapons could be split up into categories:

Good against fighters

Good against capital ships

That pretty much gives you four types of ships

fighters

bombers (fighters with anti capital ship weapons)

escorts (capital ships with anti fighter weapons)

destroyers (capital ships with anti-capital ship weapons)

With possibly another type of ship, depending on implementation

The Carrier (ship which carries/supports fighters)

In fact, you might move to drones instead of fighters, and then you can have it so that if the carrier is destroyed, the drones are destroyed or go inert.

You can get more detailed by having weapon ranges, fire rates, reload speed, some special weapons(tractor beams? stunning ion cannons?) But you can see how the systems might play out.

If the combat is auto resolve, you might look into how Dominions handles it's combat. It's very quick, but relies on some basic setup and rules for units. Unit formations, where they are placed on the battle line, and who their preferred target is. Some examples: having a forward force advancing in the center, with quick forces on the periphery, that are pegged to try to attack weak targets in the back line. Or a force that holds with some strongly armored stuff up front, and weak long ranged stuff in back.

EDIT: I missed Orymus3's post, but I'm on board with his views, and I'm a bit redundant =)

How exactly the combat mechanics of yours works? I assume it would require to track the "position" of fighters, also that they need to be "faster" than other ships (so they can attack the battleship before it incinerates everything else)? Also, you would need some "intercept" mechanic for fighters stopping fighters (also, how about you having less fighters fort protecting your battleships only while the opponent uses carriers, could these effectively protect battleships)?

Originally, I was going for an "accuracy" feel to it. Weapons have a certain accuracy (chance to hit expressed in %) and ships have a chance to dodge (also expressed in % in the form of a positive or negative modifier).

For example, a very heavy warship could have -10% maneuver, which means that a 70% accuracy weapon would actually hit it 80% of the time because it has a hard time to get out of the way. That specific bonus took into account both the expected ship's maneuverability and actual size.

But as it turned out, my combat system allowed me to replay combats in real-time and compute actual "courses" so that ships could move fast, and try to avoid projectiles, and maneuverability became much more organic. Having too many maneuverable ships, for example, won't necessarily reduce the amount of hits taken total, and you're still covering a large area of the battlemap, and if one ship dodges the bullet, another behind might just take the hit.

Kinda makes it less powerful, but I think it adds to the idea that a "swarm of Tie Fighters" would eventually get hit by quite a few missiles if only because of the sheer amount of space they occupy

But if you want to keep it simple, the % hit / dodge works just fine and reduces the need to keep track of additional information.

As it currently stands, smaller ships don't need to be necessarily faster, but it helps to close on larger foes under certain criterias. I do have a few notable exceptions however, such as a highly maneuverable saucer with no actual "helm direction" which can boost in any direction at any given time. It's not particularly fast, and it actually attacks from long range, so it can be a very big annoyance over time when massed in higher numbers.

Since maneuver minimizes the amount of hits taken, it also increases the amount of "shots lost" by the opponent, and these ships require very little armor (as they tend to effectively die in 1 shot from larger vessels).

Fighters don't need to have any specific fighter-interception mechanics. Currently, fighters are forced to move back and forth between mothership/carrier and target (to resupply and repair) which means the area in-between is a zone where fighter vs fighter encounters happen. They will both fire at one another until out of that range. Once closing on the battleship, they will evaluate the threat level. Some ships are fully dedicated at fighting other fighters (AF types) whereas others will focus on larger threats (Bomber types) and others are versatile (Hybrids). Depending on the craft and its intended purpose, they will engage accordingly. AF have higher maneuver and tend to fare better against other fighters, whereas bombers are really good at piercing shields, etc.

I'm not sure about what you meant with the carrier part, but it could be an interesting encounter. At midrange, a battleship would make short work of a carrier, whereas at longer ranges, carriers wouldn't take damage while their fighter swarm could be a real threat to the battleship. Assuming fighters survive the loss of their carrier ship (at least until the end of the fight) I would say that this would often result in draws (fighters surviving the loss of the carrier midway through the encounter, and dealing critical blows the battleship resulting in a no man's land). Economically speaking, the battleship would probably come on top given the sizeable amount of resources put towards getting fighters built, but I can see that adding a few frigates to slow down the battleship could drastically alter this (with only minimal fighter losses, all frigates, but a surviving carrier). Losing 2-3 frigates to a Battleship for the kill is a really good deal!

How your targetting system works? Does the player can decide which ship will stay in the second line? How your ships attack (target the front line always first?)

I use the same heuristics as my previous prototype. Basically, a ship is a collection of components of which weapons and helm control are not related.

Helm tends to have an objective, which is to get withing firing range of its primary target with what it considers its "offensive" arsenal only.

So in the case of a long-range ship (missile frigate let's say), the helm would consider its primary weapons to be its missile launchers only (ignoring anything else) and would move within range of a valid target to unleash missiles.

Other weapons (let's call them secondary in this case) would engage enemies as they enter a valid range, but would not have their say on the helm's control.

This makes it possible to keep ships at a distance, ready to fire with point-defense or anti-fighter weapons (gatling guns) instead of jumping in the fray.

Also I'm currently on the fence about forcing movement, that is, avoid ships being able to come to a full stop. Last time I implemented this, I was satisfied with how ships had to "arc" to keep optimal range with their target all the while staying clear from enemy shots. It felt dynamic, but in larger battles, it added some noise. I'm still hesitant about this AI feature, which is really just commenting one line of code out :P

Currently, I haven't delved very deep into targeting priorities. Originally, I wanted to have a system where components have intrinsic purpose and would tell the helm what their ideal target was and let helm decide from all available targets and ranges, but I felt it ended up breaking damage apart, with ships not focus-firing down specific enemies.

As much as I find it "boring", I tend to prefer how all ships try to focus-fire their closest enemies (often ending with 2-3 active targets on the front). Focus-fire sounds like a valid tactical approach under most circumstances.

I see that it could change based on some occurrences though. For example, high speed ships with the forced non-stop AI would end up moving "through" enemy lines and close in on linebackers organically. From there, they could inflict some damage to the artillery units.

As you can see, this specific part of the design is still up in the air (targeting AI that is) and I've playtested a number of approached, but will need to revise it as I move forward with development. It's very hard to say at the present what will bring the most fun and least frustration to the players. I do anticipate that my component approach will help me retain some flexibility in the long run to hit the right algorithm.