or more precisely (since utility systems use scoring functions, not just simple boolean evaluations):
score of option1 = f(A)
score of option2 = f(A,B)
score of option3 = f(A,B,C)

(where: A,B,C are the decision variables involved in scoring an option and are analogous to the conditions A,B,C used in a decision tree).

No. Each option is scored with all those factors included, although something in the scoring may render certain conditions irrelevant. So it's more like:

score of option1 = f(option1, A, B, C, ..., Z)
score of option2 = f(option2, A, B, C, ..., Z)
score of option3 = f(option3, A, B, C, ..., Z)

If you have some sort of explicit conditions where you never want to consider option2 when option1 is available then you're not really talking about a utility system any more. You just have a decision tree with a lot of overhead.

Incidentally this is probably the most significant problem that I have with utility systems - designers often have a mental model of the ranking or precedence that they want the actions to have, but trying to force the mathematics to produce that result consistently can be very difficult.

Incidentally this is probably the most significant problem that I have with utility systems - designers often have a mental model of the ranking or precedence that they want the actions to have, but trying to force the mathematics to produce that result consistently can be very difficult.

While this is indeed a problem, I think it's a problem with designers, not utility systems. The way designers often think about behavior (rankings of actions, if-then rules...) is brittle by nature and results in many absurd situations. Designers just need to get more sophisticated in their thinking and learn to describe their preferences as utility functions.

It is very important to give them tools to understand why an agent is behaving the way it's behaving, like a display of available actions, the contribution of each term in the utility function to the final assessment, etc.

Designers just need to get more sophisticated in their thinking and learn to describe their preferences as utility functions.

The problem is that real humans and indeed real decisions don't work that way. Usually we do have some sort of implicit ranking system but one which might get overridden by other circumstances that can be hard to predict. Trying to capture all that mathematically has been, in my experience, intractable. Even when they can see the input values and the weightings, it's not necessarily practical to be able to adjust those functions in such a way that fixes the error with the current example without breaking previously working examples. (It's actually a prime candidate for machine learning, but with that it's easy to underfit the data if there aren't sufficient inputs.)

You need to balance the scoring with all the other potential moves in mind. Otherwise it's not a meaningful comparison. Unfortunately the complexity of this obviously grows proportionally to both the number of moves and the number of conditions, so you need to find abstractions that keep it manageable. I think this is a Hard Problem. You can watch IADaveMark/ApochPiQ's video linked above to see some detailed approaches and solutions to this.

Under these conditions, to say it finds the best move is tautological - it will find the move that you have told it will be best.

The problem is that real humans and indeed real decisions don't work that way.

This is true. I have mostly dealt with situations where rational behavior is preferred, and imitating humans is irrelevant.

But I do have some experience in making utility systems that need to conform to rules. There are two ways of mixing them:
* Rules cannot be broken (e.g., you need to obey regulations in a trading system).
* Following rules is encouraged. I believe this is what applies to games.

So you can have a utility system where one of the terms in evaluating an action is whether this action is in accordance with a set of rules. The weight of this term can be adjusted so rules will be ignored at the right time. That way you may be able to get the best of both worlds: The utility mechanism will take care of emergency situations, and the designer still can describe how things should happen under normal circumstances.

For instance, a rule might say that the correct action to perform now is make tea, because a guest has requested it. But someone just started shooting. Ignoring the rules, the utility of taking cover is much higher than the utility of making tea, so even with the bonus for following rules making tea won't be selected.

The way we handle this is to stratify the utility scores. Idles, ambient behavior, making tea, etc. get a score up to, say. 1.5 ceiling. Movement and tactics get a 2-3 range score. Using combat abilities get 3-4. Emergency stuff like evasion, yelling barks, and so on might be up to 5 or 6.

The problem I was finding is that certain situations would yield low scores for combat actions and an ambient behaviour would be chosen instead, even though it was clearly inappropriate in the situation. Limiting ambient actions to 1.5 is only helpful so long as you can be sure that a combat action is always going to score above 1.5 when appropriate. Guaranteeing that can be fiddly.

I think it's important to highlight that game AI in particular is not about mimicking human thought patterns. It's about providing an illusion of intentionality and rationality.

I would argue these are actually the same thing. After all, we're not talking about mimicing neuroscience, just about making high level decisions, and it might be more reasonable to use a simpler heuristic-based approach when making an NPC do what a real person might do.