That sounds good, those damage/armor types would add depth and a sense of realism to the game. You definitely want big tank shells to be better against armored units than machine guns, and generally have the game match up to real world expectations.

The next question is how much of an effect you want the armor types to have; that is, how hard or soft the counter system will be. Realism would probably dictate a very hard counter system, where small arms would do next to nothing against tanks, but I think that tends to lead to less interesting gameplay.

The way I see it, the strength of the counter system is going to influence the player’s priorities in the game. Harder counters reward scouting, tech choices, and correct unit composition, while softer counters leave more room for micro to decide battles. Many games have a mix of harder and softer counters, but the overall balance really depends on what kind of RTS you want to make.

Large calibre Artillery works well against tanks without specifically being anti-armor (large high explosives shock waves pulverize crews right through the armor or air compression or just blowing/shattering the metal apart like it was ice). Medium and small calibre (more easily aimed) need the armor penetrating rounds.

A very basic and common start setup for RTS games is a rock-paper-scissor setup. It has a basic balancing build in and you can expand from there.

Your example could look like this:

heavy armour >> light armour >> flying units >> heavy armour

With

heavy armour: your standard tank

light armour: some kind of mobile AA

flying units: chopper

From here on you can carefully add more units, all with a certain advantage/disadvantage, like:

- Artillery: can attack long range ground targets, but although weak vs flying units

- fighters: useful vs flying units, worthless vs ground units

But always take care to not make a single unit too powerful or too useless. Some light armour type which is weak vs all kind of units is worthless and nobody would buy them. Eg an infantry seems to be quite weak at first, but if they got the advantage to be quite flexible (anti-tank & anti-air weapons) and can hide in buildings, it would make them really useful.

Typically you have vehicle and infantry armour types (machineguns good vs infantry, cannons vs vehicles). I plan to skip infantry altogether since its so hard to get right graphically:)

Would this setup make sense in an modern military/slightly scifi RTS:

If it would make sense. In modern setup no, in harder SF yes. But... even if it makes sense, would it be fun? I have played some "vehicles only" RTSes and I would say infantry is sooo cute, intuitive, distinctive, etc that you can't afford to skip them. Yes, from the designer's point of view replacing infanty with light vehicles is perfectly identical, but from the point of view of a player I have to say it sux. I feel removal of infantry is a grave digging for any RTS :(

Perhaps you should ask for advice how to make infantry graphically right instead? (maybe consider different art styles like cubes (popular now) or "no legs" like in Prison Architect or make them very, very small like a swarm of hundreds few pixels humanoid shapes?) Anyway I would not give up on infantry so easily.