Most good 4x games I've played used economy as a deterrent to large forces.

If you place 'all of your units' to defeat the enemy's strongest base, you leave yourself opened for them to overtake all of your economic settlements. You do capture one planet, but end up losing 10-20. After a few turns, your opponent has managed to use these planets in such a way that their fleet is now larger than yours.

That's the best way to do it imo.

I'm perfectly fine with concentration of forces to force a victory, because, most times, this is a bad strategy in a good 4X game...

Maybe you could add some sort of "surprise factor". If a fleet manages to attack another fleet without being detected it has advantage in the beginning of the combat, but the bigger the fleet, the higher the chances to be detected. That way, a small fleet that catches an unaware enemy can do serious damage before being counter attacked, and a succession of strike-and-disappear attacks could make a small fleet defeat the massive one.

Also, you could make enemy fleets visible only within a range of your own fleets, and the larger the fleet, the longer the distance at which you see it. So you need to have distributed fleets if you want to detect approaching enemies from all directions, and use small fleets if you want to avoid your enemy knowing where you are.

Another idea you could implement is making movement of fleets expensive. If you have one huge fleet going from side to side of the galaxy, well... you're gonna run out of resources pretty fast, so if you want to be able to fuel your ships for the whole war, you better have several fleets distributed in manageable regions.

EDIT:

The second biggest problem I see are meaningless planets. After the war starts they become useless (not enough time to build ships) while ships became the only variable that counts.

This brings me to Paradox's games, which have various interesting mechanics. Like a province can have only half soldiers drafted into an army and the other half is obligued to stay as garnison. Also, conquering a province takes a lot of time even if the attacker have a big advantage in numbers. In short, a province can defend itself effecively for a significant time *without the player investing anything* in defence. I don't think these really fit the space 4X theme but still as a mechanic it works.

Well, you can came with some mechanics to make planets useful. During a war, militar factories have a lot of work, after all. Maybe the biggest ships take too long to be build during the war, but there can be smaller and cheaper ships that can be build fast enough, also the planets can be important for providing fuel and reparing existing ships. And you don't want your enemy to capture your fuel reserves while your only and massive fleet is too far away to counter attack, do you?

In Master of Orion II (one of my favourite games btw), battles take place completely within one "galactic" turn. Also there are always option to destroy a small ship without it ever being able to do anything.

I always wondered if it was possible to change the rules in such a way, that battles are not concluded within a single galactic turn (with either one fleet completely destroyed or retreated), but instead have battles last for several galactic turns (stopping after a number of battle turns and continueing in the next galactic turn). Also switching the active player after a number of unit-turns instead of after he has used up all his units' turns could make battles more balanced. That way, it would be possible for either player to bring new ships into a battle, smaller ships could destroy bigger enemies even if the enemy has the advantage of the first turn.

Also, stronger area of effect weapons that can be placed on smaller ships could help. There's for example the Pulsar in MoO, or the Spatial Compressor. Having some of those be actually more powerfull (not only more efficient) against larger fleets could counter the huge-fleet-syndrome. The strength of those weapons could also be an effective measure to balance the game towards a certain maximum sensible fleet size.

I like the other suggestions about fleet being more stealthy if they are smaller very much. Small fleets could even be made to pass through planetary defences more easily and probably even start their combat between the defending fleet and the planet if they are very small (sneaked through the defending fleet), thus allowing them to drop ground troops faster and so winning the battle without actually defeating the defending fleet.

What if ships had an upkeep system similar to Kohan? All ships take X resources PER TURN, as opposed to a one time cost. Each planet produces some X per turn, so losing a planet immediately hurts, and you can go quickly negative. If you go negative in X, ships lose health, go slower, or what have you. It allows for a counter against a super stack by splitting ones forces and conducting small raids, especially if one can get behind enemy lines to planets that might produce a lot, but not be well defended.

ferrous: Master of Orion has something similar and it didn't stop players from stacking up. In MoO, you have command points (not sure if that's what they're actually called), which are produced by certain buildings of which you could only have one per colony. Bigger ships require more command points. If use up more command points than you create, the maintenance costs of your fleets are increased quite a lot actually. But with just a few economically strong colonies you can counter the effect by producing trade goods.

I guess Microprose had the same idea about limiting the usable number of ships, but it didn't work. At least not in the way they intended. In the early stages of the game this is sometimes a limiting factor to the speed of fleet growth but in the long run, new technologies and economic tactics make it a useless trick. Maybe that was actually done intentionally as some fort of positive feedback mechanism to make games end in a reasonable time (even single player games can take up a whole day though).

ferrous: Master of Orion has something similar and it didn't stop players from stacking up. In MoO, you have command points (not sure if that's what they're actually called), which are produced by certain buildings of which you could only have one per colony. Bigger ships require more command points. If use up more command points than you create, the maintenance costs of your fleets are increased quite a lot actually. But with just a few economically strong colonies you can counter the effect by producing trade goods.

I guess Microprose had the same idea about limiting the usable number of ships, but it didn't work. At least not in the way they intended. In the early stages of the game this is sometimes a limiting factor to the speed of fleet growth but in the long run, new technologies and economic tactics make it a useless trick. Maybe that was actually done intentionally as some fort of positive feedback mechanism to make games end in a reasonable time (even single player games can take up a whole day though).

That's just poor implementation, in Kohan, if you lost your major economic cities, you were truly horked, unable to build, and unable to repair your troops, and they would take repeated damage if they left your supply zones.