Well, I'm not an expert in AI and several things are quite new (in relation to 11 years of development), but I've learnt a lot about AI in the past year.
The whole system is build up of several AI related components which are displayed in the figure.
The most basic component is the finite state machines (FSM). I use several FSMs in my game engine, mostly for controlling entities, but although for gui and user interactions. They are quite simple, communicate asynchronous over a central bus system and can be designed in an uml tool for which I wrote a xml converter. I can attach several FSMs to an single entity to control it. Most dynamic agent got at least two FSM, one for controlling the actual movement-action and one for controlling decisions and reactions (think of body and mind).
The second basic component is the way-point system. Instead of using a navigation mesh, I developed a multi-layered way-point system. Currently only two layers are needed, whereas one is for mostly navigation and the second layer is at a lower resolution containing additional meta data.
The third basic component is a knowledge container, containing knowledge about locations (represented by way-points). There's no special knowledge data structure about entities or events. The knowledge contains data about explored, dangerous, inaccessible, or safe locations. Each entity has its own knowledge, but certain entity group are able to share knowledge.
There are two different "traditional" AI systems which works on top of the three basic components. This is a behaviour tree which controls the behaviour of a single entity and a blackboard system for a high level orchestration of entity groups.
The blackboard system manages jobs. Each entity is able to create certain jobs( like "supply me with food") or to apply to one job. I use the blackboard system to control group of entities in a economical sense. In dungeon fortress a group of entities, like a spider nest, is a more or less simulated community of entities. They need to eat, to gather resources etc.
The control of a single entity is done by the behaviour tree which is inspired by the work of Chris Heckler, who uses a similar system in spore. A behaviour tree, is a tree of "decision" nodes. Each node is quite simple, but by combining this nodes in a tree like structure you can create more complex behaviour with ease( I really love behaviour trees!). The behaviour tree
make decision based on different entity properties, by scanning the surrounding for enemies, resources, objects of interest and eventually by a modified A*.
The A* is most likely in every game AI system present. I used a modified version of it to navigate on the way-point system by using the knowledge of the according entity.
This way it is possible to navigate around dangerous areas or to avoid blocked passages.
