I'm working on a top-down squad base shooter where the player controls the squad leader and can also issue commands to other teams within his squad.
- Soldiers respond well on their own, reacting to and defending themselves, taking cover, assessing dangers and tactical positions, etc.
- Teams(3-4 soldiers) move cohesively, but are not just a square blob of men waiting to be blown up.
- Teams should also be able to receive orders to accomplish an objective (provide cover, move here, take building, etc.).
- Problem solving: if a player tells one team to enter a building but the door is locked they should look for other ways in and act accordingly, preferably taking into account the current combat situation. (if the squad is not in combat, don't open the door with C4... If we are in combat, don't waste time searching a huge building for an unlocked door...)
I have looked at dozens of tutorials on both Behavior Tree's and Finite State Machine's and they seem like they both could accomplish the task, each having it's own advantages and disadvantages. I have also implemented both and have prototyped some of the basic behaviors in each.
With FSM's, I can easily reason and understand the states and transitions involved, but can see the required number of states exploding quite quickly and it seems that every time a state is added I have to spend a lot of time checking the transitions and routes through the machine to try and recognize any problems. Simple to reason with, hard to extend.
With BT's I struggle to organize the tree's execution in my head, and the code to define the behavior tree is too abstract for me to build a working mental image of the tree, but I can very easily add/remove behaviors from the tree without having to change other nodes. Harder to reason about, easy to extend.
I am leaning towards BT's because of their extensibility, but I think i would need to implement a graphical editor in order to effectively reason about them. This wouldn't be too huge of a task - I have something very similar already put together for another project and so I expect it would only take a few hours time to create this.
Does it seem like I am doing something wrong?
Are neither of these the right answer?
Is the BT editor worth it, and out of curiosity, does anyone work with these exclusively in code?