If i remeber correctly, you want to keep things rather abstracted/simplified.
Why not have an efficiency factor. Starts at 100%. At some point a stack becomes harder to handle and efficiency softly drops, so it makes sense to not make stacks of doom.
navy size efficiency navy combat strenght
<100 ships 100 % 100
150 ships 85 % 127
200 ships 70 % 140
250 ships 55 % 138
300 ships 40 % 120
>350 ships 25 % 88
You get the point, this needs tweaking of course but the point is that its a simple system that the player can easily moniter.
It makes it better to have several small/medium sized fleets than one huge.
If you have that level of complexity the area of effect weapon idea is good. It counters big stacks efficiently if it deals dmg to all ships in a stack, no matter how big the stack is.