Very interesting. So you're AI queues up an "Update Path" job to your pathfinder processor and enters itself into a "wait for path update" state so it can continue to monitor things. Then before the pathfinder thread rejoins, it fires off an "Update AI with Path" event which the AI consumes at which point it changes itself back to some sort of action state? Very slick!

Yes, the same principle (I'm using BHTs, nost only FSMs, which can take control from the waiting node ).

Are the events managed via the AI or are they managed via the pathfinder processor, or some game event manager?

I have a general purpose bus system in my engine over which several components can communicate via events.

Space partitioning - simple is 2D 3D zones that evey movement removes only zone registration and adds to new or leaves the same if unchanged
A grid square or cubic area with size about that of the maximum interaction range (or finest LOD). Since you usually wont be centered you will have to check all
adjacent squares (8) /cubes(26) and filter with true interaction distance to get a culled interaction/sensor list for the object. If populations are usually low per 'zone'
then simple fast link-lists can be used instead of a more heavyweight (and wasteful) container types.

Individual objects maintain a 'target' list of already assessed 'in-range' targets so that you dont have to reevaluate them every time.
You will have to add newly seen targets and remove long non-visible ones AND if your AI logic evaluated actions or distances
(ie- closer is a higher threat). Thats Delta (change) processing that would require reevaluation due to some difference in the situation)
AI interpretation of the situation dwarfs the basic object culling processing so if you can cache the results of alot of classification
processing you can often save alot of CPU resource. That also allows for AI processing to have a continuum of identified objects to
watch patterns of behavior (and depending if you have partial/faulty sensor info to infer positions of somethig you cant temporarily see)

If your game has 'sides' then the results of AI classification/assessment might be pooled per side instead of per object. Grouping entities
(ie- a fleet) pooling at a smaller size than a 'side' likewise might be done.

LUA runs about 10X slower than native code so if you can move some of these bulk crunching operations to C++ then you can save
a significant amount of CPU and leave more time for the irregular logic of the scripting.


I think LOD processing interval was already mentioned (long range scan would happen less often than close range where the more
complex/immediate/high-priority interactions will take place.

Have you actually profiled your code at all?

Have you actually profiled your code at all?

Yes, I have an ingame time profiler on microsecond base to measure the avg call duration and the call count of certain important processes. The benefit is, that I got time measuring on every run with a detailed log at the end, I can narrow down on certain test cases and I can use it within lua. Additionally I use sleepy, a free profiler.

I have not used any commercial profiling tools sofar, I save the trail versions for hardship cases and I've tested the lua profiler without success.

If you already have profiling data, any clear optimization wins should be staring you in the face.

Also, without us having access to that data, pretty much any advice you get on optimization here is going to be about as useful as blindly pounding on the keyboard and hoping a faster algorithm pops out when you're done. We don't know nearly enough about your systems, your code, your hardware, or anything else relevant to your situation to be able to give you really relevant advice on your particular case. The more you can share, the better.

If you already have profiling data, any clear optimization wins should be staring you in the face.

Also, without us having access to that data, pretty much any advice you get on optimization here is going to be about as useful as blindly pounding on the keyboard and hoping a faster algorithm pops out when you're done. We don't know nearly enough about your systems, your code, your hardware, or anything else relevant to your situation to be able to give you really relevant advice on your particular case. The more you can share, the better.

Yes, it dawns on me that this special case seems to be more of a monolog where I must help myself (and yes, I'm currently optimizing the bottleneck staring right at me ). The issue is, that I need quite "advanced" advices to optimize it further. The engine is in development since 12 years and it is far from perfect, but I've done a lot to keep a clear design, to optimize performance and keep it stable and bug free. It is quite teethed, writing all this down to give enough information would lead to a wall of text.

Ok, an other, more general approach: How to make an AI written in lua multithreaded ?
Don't answer this question yet, I think that I need to create a new topic asking this question. Sofar I'm thanking all who tried to help me arranging my ideas

I arrived a little late for the party, but on the individual vs swarm AI, I would submit to you that when you have so many agents clustered together it would actually be desirable for them to team up and behave as a group, even if they started as individuals, if two stray army soldiers on the same team meet up in the battlefield they would likely join forces as long as they have a common goal, in this case, killing the player.

As for lua scripting, make sure you use lua to connect the dots and not to make the dots, leave calculations and behavior implementation to the native language and use lua only to trigger the reactions to specific events, events that should also be calculated and triggered from native language.