So I am working on a 2D game in Unity that would probably be best described as a top down game similar to Starbound / Terraria with a much heavier focus on combat and character progress and less on the exploration / building.

Up until now my enemy controller has been very basic with just inline code that does basic checking around itself for the player, chasing the player if in a certain range, attacking the player in a certain range, and so on. I am now trying to have more behaviors that the enemy can have and having all this inline in the enemy controller is getting a bit hard to manage and make more modular to make enemies with different behavior easily. For example I would want to have a number of different actions an enemy can do:

• Roam when a target is not in range (whether the be the player or another NPC)
• Roam when a target is not in line of sight
• Chase target when it enters within the enemies range
• Chase target when it enters the line of sight on the enemy
• Range attack target when it is in certain range of the enemy
• Melee attack target when it is in certain range of the enemy
• Teleport to a random location within a certain range when the target gets close enough to the enemy
• Perform seal heal when enemy's health gets too low
• Run away when the enemy's health gets too low

Not all enemies will have all these possible actions so I want to be able to as easily as possible add and remove these action for enemies and just as easily add new actions to the pool of available actions.

I am trying to figure out the best way to handle this kinda of behavior. Finite State Machines seem to be the most common thing I see suggested but was wondering if there might be something else I should be looking into.

Also looking for learning resource for whatever solution I end up going with (ideally something in video form and even better if it is focused on a C# implementation).

Any help with this would be awesome, Thanks.

I have a similar problem for my game. Don't go with an FSM, unless you are ready to sink alot of time into writing or learning a dedicated FSM library. Your behaviors are way to complex and numerous to be implemented in a simple had hoc FSM, the code will end up harder to manage than a bunch of inline and nested if statements. I came up with an elegant and very workable solution, sort of my own interpretation of a utility style AI.

First thing I did was to separate the AI actions from the AI behaviors (in this context behavior = decision making entity). Basically each AI units have a set of behaviors, which each have an evaluate method and a run method. At every N update, the AI unit will run the evaluate method for each of it's behavior, and will run the one with the highest score. The run method contains the rules to modify the action list of the unit. The action list is a queue of actions ranked by the order in which they get executed. An action might take any number of turns to complete or fail, and it contains the actual code that acts on the unit (move, attack, etc). Of course, the behavior can get instantiated for a particular unit with a multiplicator, which multiply the score of the evaluate method, which allows for fine tuning of behavior to make some units more aggressive than others. Also, just be clear, the evaluate method takes the environment and the state of the unit into account. For example, to evaluate the "heal self" behavior, the code would probably look like this:

( (unit.health.current - unit.health.max) * -1 ) + (distance(unit.position, player.position))

Of course you can get crazy and make all kind of weird functions to get the desired effect.

This method allows you to write every behaviors only once and use composition to quickly make new mobs type. It also separate the actions from the decision, so a decision may result in a queue of actions. Multiple behaviors might share multiple actions. I don't know if this is the *best* way to do this, but it works pretty damn well and even I am impressed by how great it is.

What you described is part of using the state machines, and nested FSMs are the standard way of doing this.  

You cannot account for everything in a large game with a single state machine. There is one or more additional layer in choosing which state machines to run. 

State machines are expressed in terms of behaviors like you described. Often this is done by passing in the character's driving motives and running it through a function to determine the utility of the behavior.  Often the utility functions require a source, an object, and a target.  For example, this character to use a medkit on their neighbor, which is a different utility than this character to use a medkit on themselves. A character that is hungry may derive a high utility score from a task called "eat".  Another character may derive high utility from a task called "use medkit".  Another may derive high utility from "use sniper rifle".  Those behaviors ARE families of state machines.

Once you've got an action in mind, run the state machine. The state machine for a medkit in your inventory may be short: hold the medkit, play an animation, adjust health, destroy medkit.  The state machine for a sniper rifle may be more complex: find a good sniping location, travel to that location with another state machine (with the option to re-evaluate behaviors), once in location verify that you can take the shot, wait for a good shot (with the option to re-evaluate behaviors while waiting), then take the shot when done.

As you build the state machines they quickly gain more layers of nesting.  Travel to location can require a sub-task of opening a door, or a sub-task of using a vehicle, or a sub-task of riding a dinosaur.  Each sub-task might further have additional sub-tasks to run. Each one a FSM, nesting as needed until the ultimate task is complete.

Though I might be mistaken, I thought Finite State Machine had the formal attributes of :

• Containing the code to change state within the states themselves
• Be strictly able to change state through pre programmed 'transition'
• Lack any kind of flexibility regarding on the fly behavioral changes

When I hear FSM, I think about the following :

(image from https://gamedevelopment.tutsplus.com/tutorials/finite-state-machines-theory-and-implementation--gamedev-11867 )

Wouldn't this be hellish to implement in OP's context, where he wants "to be able to as easily as possible add and remove these action for enemies and just as easily add new actions to the pool of available actions."

My attempt to implement a finite state machine was a monumental failure. To be honest, anything more than 5 states becomes very hard to manage in my experience, and you will likely have to produce documentation in parallel to your code to keep track of everything. Just thinking about sitting down and drawing op's fsm is giving me a headache. Take my advice with a grain of salt, more so because Unity is bound to have tools to easily create and manage fsm, but from a C# coding perspective, I advice you to reconsider your options.

Good writeup. The Sims always seems to be a go-to example. State machine on top of state machine on top of state machine.

In TS2 the state machines were driven by the Edith tool, and scoring functions were a bit of a pain. Those in turn were driven by code that was boilerplate and boring to write. TS3 had simpler graphs in Jazz, yet the tool added far more flexibility to the state machines. I loved that state transitions could be probabilistic, designers and animators didn't need to ask programmers to put them in any more, they could adjust the odds easily. The code side was much easier to write, I could build a new Sims interaction with all the ranking functions and basic steps  through the code-side FSM tasks (route the Sim, drop an animation jig, re-verify validity, walk through the Jazz FSM, clean up) in about 45 minutes, such an improvement.

I preferred Jazz graphs over the Unreal blueprints, although those and Unity's animation FSM graphs are the closest most developers experience. 

@FFA702 a state machine is a concept. We have many great articles on the site regarding AI behaviors driven by state machines. I would add links but I'm on mobile where links are difficult so you'll have to search them out. Many describe how we AAA developers worked on assorted FSM tools to drive all the action. I have used them extensively in many games, The Sims 2 and 3, the Littlest PetShop series, Fortnite, and every minor game I've worked with. They are how games do AI. Behavior trees help choose which FSM is run, but the details are all about FSMs.