I try to design a combat system for a 4X that has these characteristics:

- the map is made of systems/planets connected by starlanes (so predefined paths/connections)

- the player moves several fleets (not a single big one) to defend/attack

- the key is the control of territory (system), once you are there they can't get rid off you too quickly (also there are choke points and you can't "teleport" behind "enemy lines")

- there should be relatively low or moderate loses (not all ships lost in one big battle in a single turn), so there need to be some retreat mechanic (the goal of a battle is not to destroy enemy fleet but to force them to retreat from the system)

Ideas how such mechanic should work?

I'm not that familiar with tactical hex boardgames :(

The part I need/want the most is no total destruction of a formation (fleet) ever (even if zero ships left - it's just a bunch of escape pods then). So, I prefer if the auto retreat always work (no overrun, no annihilation after X retreats). Maybe make two rules of movement, one when the player moves the fleet (restrictions) and one when it retreats (no restrictions, can enter any system, etc)? But it could cause "skipping behind enemy lines" in some cases...

Or maybe when a fleet is forced to retreat and there is no legal place to retreat the fleeet enters "return to base mode" and auto move to the nearest miliary base using explored systems only as route and with disabled scanning on the way (in case of moving through enemy territory) and the player regains control over it only after it reach destination?

When you need to see some crazy ideas for board game checkout the Kickstarter board game area. The videos are often short and you can see a lot of ideas in small amount of time.

You may want to pick up a book like this one: http://www.amazon.com/Simulating-War-Studying-Conflict-Simulation/dp/1441185585

I must second that suggestion -- a fantastic book that applies to both board games and computer games.

I've likely mentioned it to you before, Acharis... you really, really should test out these ideas in Machinations:

http://www.jorisdormans.nl/machinations/

It is SO, SO useful. If you need any help getting used to the interface I'd be glad to help. Look in the tutorial projects for some inspiration -- I think some Starcraft mechanisms are plotted out in there.

Anyway, I don't think boardgames are the answer (all I have seen have destroy if can't retreat which is not what I could use).

And what you think of the "return to base mode" mechanic? SHould I go for it? Or do you have a better concept?

I've likely mentioned it to you before, Acharis... you really, really should test out these ideas in Machinations:
http://www.jorisdormans.nl/machinations/

Anyway, I don't think boardgames are the answer (all I have seen have destroy if can't retreat which is not what I could use).

Because you've seen EVERY board game ever, right? I suggest you take a broad approach to gathering ideas. And I guarantee you haven't seen Combat Commander.

I'm a programmer, it's far more convenient for me to just code it quickly in PHP

As the Dead Kennedys said: "Give Me Convenience or Give Me Death."

I think you've missed my point AND the tutorials (which are full of examples of ideas/concepts/frameworks). Not your cuppa, I suppose.

If you can code it quickly, why not just implement all of your ideas incrementally? And I like "Return to Base" as a concept -- some sort of way to regroup that makes sense in the setting (I can't remember -- is there FTL travel?).

"Programmers deisgning, designers programming, dogs and cats living together... mass hysteria!" -- Dr. Venkman

Ignoring all the distractions (inspirations, websites, cute design tools and the like) I will pursue the mechanics the old pen & paper style (or keyboard & forum since that's what is used nowadays) :D

So, back to the mechanics (reminder of the desired characteristics):

- the map is made of systems/planets connected by starlanes (so predefined paths/connections)

- the player moves several fleets (not a single big one) to defend/attack

- the key is the control of territory (system), once you are there they can't get rid off you too quickly (also there are choke points and you can't "teleport" behind "enemy lines")

- there should be relatively low or moderate loses (not all ships lost in one big battle in a single turn), so there need to be some retreat mechanic (the goal of a battle is not to destroy enemy fleet but to force them to retreat from the system)

Rules of moving: (need polishing and clarification)

- you can move your fleet up to 4 systems as long as the path is free of any enemy fleets

- if you move to unexplored system the fleet stops (ends turn)

- you can move to a system occupied by enemy if you move from a system not enganged in battle (no enemy or enemy retreats/broken)

- you can move from a system with enemy fleet only to a system that is under your control OR to anywhere if your fleet is at least 20 times stronger (no single ship stopping an armada)

- anytime you can click "return to base" button (confirmation will be given), a while flag appears on your fleet and it auto retreats to the nearest military base ar a speed of 3 systems per turn ignoring all enemies, not scanning anything, etc (ghost mode), then it needs to stay in the miltary base for additional 1-2 turns; also that fleet gets a small morale loss

Rules of combat:

- when two opposing fleets meet in one system a battle occurs at the end of turn

- the battle continues each turn until one fleet moves away (if possible), retreats willingly or is forced to retreat

- when one fleet is much weaker it's forced to retreat after just one turn of battle, it also gets a significant morale loss

- during battle ships can be destroyed (not that often), heavily damaged, light damage, not damaged

- the fleet moves and retreats as a whole (single ships never leave fleet or move alone)

Rules of reinforcements:

- ships within a fleet can slowly repair on their own (bonuses possible)

- when a fleet is not moving and/or is orbiting a friendly planet the repairs are significantly faster

- heavy damaged ships are repaired only when the fleet is orbiting a system with a military base or a shipyard

- newly produced ships can be "teleported" to the fleet as they are produced, but there is a limit how many reinforcements can a fleet recieved per turn (if in battle almost none, if orbiting a shipyard unlimited)

A defining feature of space is its vastness: any number of starships, with several orders of magnitudes of margin beyond the expected maximum number of ships in the whole game, should be able to occupy a map sector without being engaged in combat.

Combined with the preference for "starlanes" (suggesting that everyone hangs around their endpoints as much as possible) and the desire for protracted combat, the possibility of coexisting with the enemy in the same location suggests rules in which engaging the enemy is an optional and often deliberately delayed action, and choosing where to put reinforcements and who/what deserves attacking are the main strategic elements.

• Every map sector contains any number of mobile fleets and stationary planets and stations of any faction. Newly built ships join a fleet in their sector at the end of the turn and can move the following turn.
  Sector size can vary from most empty and a few with one planet or station, to planetary neighbourhoods encompassing satellites and Lagrangian points, to whole solar systems; something important to vary and tweak for tactical and strategic purposes.
• Movement speed: In a turn, a fleet can attack any enemy fleet in the same sector (or merge with a friendly one), or go to a planet or station. Fleets can split freely at the end of the turn (to show the resulting new fleets when the enemy selects moves next turn).
• Fleet combat takes place after movement; it can be assumed that all fleets who go to the same place or approach the same fleet arrive together. With suitable cultural excuses (prudence, chivalry, expending limited fuel or ammo, etc.) the normal outcome is that both factions retreat after modest damage, it doesn't matter where. Next turn they'll go anywhere in the sector all over again.
• When a fleet goes to an enemy planet or station, they first engage enemy fleets who went there (and those who went after them); if attackers are successful enough, as an exception to the rule of always retreating, they remain, forcing the defenders away, and attack the objective in the same turn.
• "Long-distance" movement: a fleet shouldn't go further than to an adjacent sector in a turn, and only if no enemy fleet attacks them. On the following turn, they are in the other sector.
  Spending additional turns in wormholes, warp speed, or whatever you want to call states in which the fleet is traveling and cannot attack or be attacked can be an option.
  Slow movement enables bluffs; fast movement wouldn't require a player to commit its forces to a specific place. Slow movement on a large map can be made more agile with some kind of shortcut between ordinarily distant sectors.