My game's a 3D starfighter project with a sort of unique AI. Internally there are some heirarchical state machines that determine things like whether the enemy is attacking or retreating, but really the important part is the enemy must choose a direction to fly at the end of the equation (towards the player in an attack scenario, or away from the player in an escape scenario).

To determine the direction a weighted sum equation is used currently, roughly based on the boid algorithm to produce flocking behavior, of the form:

boidDirection = weightAvoidance * avoidanceDirection + weightAttack * attackDirection + weightAlignment * alignmentDirection + weightCohesion * cohesionDirection

While this AI seems to excel in 95% of scenarios, it fails miserably when the enemy is trapped between the player and a surface because I'm apparently doing a lousy job of determining the avoidance direction vector.

The current formula I use to calculate the avoidanceDirection vector is:
1) The enemy knows the position of other nearby ships and uses explorative raycasting in a forward facing cone to "look" ahead for collision avoidance with more complex geometry.
2) It then tries to avoid all of these things by calculating the direction it should travel as the sum of vectors away from each nearby ship or raycast intersection (weighting these avoidance vectors relative to distance). This can yield a different direction every frame as a result of new raycast discoveries (causing the jerking behavior described below).

This works well, except in the scenario where the player chases an enemy toward a "wall" (e.g a flat or concave surface on the hull of a much larger ship), and once the enemy "sees" the wall, it jerks around indecisively as it tries to find a clear direction to turn, but it's raycasts just show more wall ahead everywhere it looks. After jerking around indecisively for a few seconds the enemy comically flies into the wall because it couldn't find a good direction to turn.

This diagram essentially shows what this looks like from the perspective of the enemy. It sees things to avoid everywhere in front, but it must head towards all those things for a short while as it changes direction to get to the safe direction (green). As the enemy turns towards the chosen vector (green), it sees more raycast collisions in that direction, causing it to choose a new direction to go because those new raycasts suggest it should choose a different path (indecisive behavior).

Any suggestions for an algorithm that would choose the green vector consistently and not indecisively change direction?

Finding: using the surface normal rather than avoidance direction for the raycast intersection points solved the problem in the scenario above.

However, it still seems to perform poorly in my tunnel scenario because the weighted sum results in the enemy to randomly turning towards the wall where it has had the fewest raycast intersections, resulting in a similarly jerky behavior until it eventually crashes into a wall.

Consider the scenario illustrated by the following diagram. The enemy has done 3 raycasts, while what we want is an avoidance vector like the green vector, instead the enemy will point straight towards the wall for which it has only done one raycast (because of the weighted sum). It seems that the problem lies with my use of a weighted sum of avoidance vectors, but I'm struggling to come up with a better algorithm.

I believe in theory what I want is the vector v that maximizes the following equation (where A is the set of all avoidance vectors), but I'm struggling to think of an efficient way to solve for that vector.

[sharedmedia=core:attachments:12433]

The only solution I can think of basically amounts to choosing random vectors v and testing them with the equation. This doesn't seem particularly optimal since it would have to be done for each enemy ship.

Perfect, thanks!