I'm working on a top-down squad base shooter where the player controls the squad leader and can also issue commands to other teams within his squad.

My Requirements:

• Soldiers respond well on their own, reacting to and defending themselves, taking cover, assessing dangers and tactical positions, etc.
• Teams(3-4 soldiers) move cohesively, but are not just a square blob of men waiting to be blown up.
• Teams should also be able to receive orders to accomplish an objective (provide cover, move here, take building, etc.).
• Problem solving: if a player tells one team to enter a building but the door is locked they should look for other ways in and act accordingly, preferably taking into account the current combat situation. (if the squad is not in combat, don't open the door with C4... If we are in combat, don't waste time searching a huge building for an unlocked door...)

I have looked at dozens of tutorials on both Behavior Tree's and Finite State Machine's and they seem like they both could accomplish the task, each having it's own advantages and disadvantages. I have also implemented both and have prototyped some of the basic behaviors in each.

With FSM's, I can easily reason and understand the states and transitions involved, but can see the required number of states exploding quite quickly and it seems that every time a state is added I have to spend a lot of time checking the transitions and routes through the machine to try and recognize any problems. Simple to reason with, hard to extend.

With BT's I struggle to organize the tree's execution in my head, and the code to define the behavior tree is too abstract for me to build a working mental image of the tree, but I can very easily add/remove behaviors from the tree without having to change other nodes. Harder to reason about, easy to extend.

I am leaning towards BT's because of their extensibility, but I think i would need to implement a graphical editor in order to effectively reason about them. This wouldn't be too huge of a task - I have something very similar already put together for another project and so I expect it would only take a few hours time to create this.

Does it seem like I am doing something wrong?

Are neither of these the right answer?

Is the BT editor worth it, and out of curiosity, does anyone work with these exclusively in code?

So, taking your input into consideration, here's my rough plan:

Spatial reasoning is mostly being done during level design right now - at runtime soldiers are just rating the positions based on the current situation (angle of incoming fire for example). I tried working on some algorithms to calculate points of interest automatically, but it was seeming easier to just manually place them during level design (for which I built a graphical editor).

FSM to control squad level tactics. A squads number of states is likely to be far less than that of an individual soldier, therefore a FSM should model their planning well without ballooning into a tangled mess of transitions. Also, a squad state is going to be tracking much more world data (soldier positions, other team positions, routes, etc) and it will be easier to manage that data in a state machine than a behavior tree.

Behavior trees for soldier level actions - BT's seem very reactive and fluid, and scale much better to a lot of fine grained actions than a state machine.

Squad level tactics could be sent to soldiers via their individual blackboards, but I think it would be easy enough to build some kind of hybrid data structure available to the squad and it's members where a soldiers requests to the blackboard are automatically scoped to their own set of knowledge - either data they put in the board or data that was set by the squad manager... Hmm, fun programming problem.

Thank you for your input - it helped me solidify my plan quite a bit.

Sorry to possibly throw a wrench in your plans.

I have exactly 0 experience in making an AI, but have been reading about various tactics in implementing things, as they look interesting.

I know FSMs (but in a different context), and browsed BTs a bit (and like you I have problems picturing its execution).

Recently I ran into GOAP (goal oriented action planners). In GOAP, you let go of rigid order in decision making, and instead have individual actions that you combine into achieving a goal.

I cannot tell you how this compares to FSM or BT, or even if it actually works, due to my 0 experience, but it sounds like this may be an alternative that could be of use, for some part of the decision making.

As Sean say's there is no hard either/or choice to make, FSM and BT are inherently mixable. On the other hand, I have shipped titles with only FSM's and others with only BT like approaches. When I have mixed I usually break things down into two groups, items with well known and fixed rules defining entry/exit/transition use FSM and items where the rules are a bit more sloppy become BT's. For instance, a relatively common FSM state: moveTo. It moves the actor from point a to b and exits only if the target has been changed or reached, an interrupting event such as death, decision interrupt (i.e. a new enemy shows up in target area) occurs etc. Generally you code such a thing up and reuse it in lots of places and it is a tool you don't have to tweak a lot. On the other hand, the decision to call moveTo is often based on a list of priorities which BT's make fairly easy, i.e. if I'm low heath and that's a good hiding spot, moveTo, or I'm low health and in cover, don't moveTo, etc.

Getting into the more specific of your goals though. Generally speaking FSM versus BT does not solve for the squad tactic side of the equation. In this area you generally need an invisible actor which is a representation of the squad and handles the overall intelligence. Generally speaking this means turning off the individual unit decision making and letting the invisible object be a general planner. In the past I have done this using multiple levels of FSM and BT where the squad intelligence says things like move into this area which spawns a FSM which moves the individual units to new points in the general area. The top level intelligence gets a notification when moved or some interesting thing has happened and starts planning the next thing for the squad to do.

The hierarchical behavior systems as mentioned are easy to describe but often a nightmare to implement. So, rather than worrying so much about the one or other choice, I'd suggest you look into planning how you would do something simple like having a squad go to a location and take cover with all the little edge case rules needed such as two units don't try for the same cover, the individually go to their closest cover, sniper units stay to the rear of expected area of action, etc. Getting that figured out in a hierarchical or other AI system is a massive pain in the ass but when done correctly looks awesome to the end player that watches "smart" behavior of enemies.

Hope this gives you some more ideas and refinement.

I know FSMs (but in a different context), and browsed BTs a bit (and like you I have problems picturing its execution).

Recently I ran into GOAP (goal oriented action planners). In GOAP, you let go of rigid order in decision making, and instead have individual actions that you combine into achieving a goal.

Thanks for the suggestion, I've read about GOAP before and was planning on exploring them with the squad level actions.

As Sean say's there is no hard either/or choice to make, FSM and BT are inherently mixable. On the other hand, I have shipped titles with only FSM's and others with only BT like approaches. When I have mixed I usually break things down into two groups, items with well known and fixed rules defining entry/exit/transition use FSM and items where the rules are a bit more sloppy become BT's. For instance, a relatively common FSM state: moveTo. It moves the actor from point a to b and exits only if the target has been changed or reached, an interrupting event such as death, decision interrupt (i.e. a new enemy shows up in target area) occurs etc. Generally you code such a thing up and reuse it in lots of places and it is a tool you don't have to tweak a lot. On the other hand, the decision to call moveTo is often based on a list of priorities which BT's make fairly easy, i.e. if I'm low heath and that's a good hiding spot, moveTo, or I'm low health and in cover, don't moveTo, etc.

Getting into the more specific of your goals though. Generally speaking FSM versus BT does not solve for the squad tactic side of the equation. In this area you generally need an invisible actor which is a representation of the squad and handles the overall intelligence. Generally speaking this means turning off the individual unit decision making and letting the invisible object be a general planner. In the past I have done this using multiple levels of FSM and BT where the squad intelligence says things like move into this area which spawns a FSM which moves the individual units to new points in the general area. The top level intelligence gets a notification when moved or some interesting thing has happened and starts planning the next thing for the squad to do.

The hierarchical behavior systems as mentioned are easy to describe but often a nightmare to implement. So, rather than worrying so much about the one or other choice, I'd suggest you look into planning how you would do something simple like having a squad go to a location and take cover with all the little edge case rules needed such as two units don't try for the same cover, the individually go to their closest cover, sniper units stay to the rear of expected area of action, etc. Getting that figured out in a hierarchical or other AI system is a massive pain in the ass but when done correctly looks awesome to the end player that watches "smart" behavior of enemies.

Hope this gives you some more ideas and refinement.

Thank you very much for your suggestions. Regarding the invisible actor - that's exactly how I was thinking about it actually. I hadn't thought to spawn an FSM (or even a BT) for a certain goal however, and will definitely explore that.

When I said I was going to use an FSM for squad level actions, I definitely wasn't thinking it would be the golden ticket that would solve everything, and I might not even implement it as a typical FSM, but I think the general concept of states with well defined transitions between them lends itself well to the high level structure of a squad AI.

Definitely just need to start putting stuff together.