1 hour ago, Michael Aganier said:

I'm not very familiar to how other games do it

Other games don't try and simulate 500,000 units - more like a thousandth of that, which means a million times fewer potential collisions to check for. That's basically the only really important fact here.

1 hour ago, lawnjelly said:

 the bigger question that comes to mind is why you want to run AI / physics / individual rendering on 500K units?

It's really a matter of pushing the optimisation as far as I can. I don't want to lose the physical feeling of each individual unit. In the end, if I can get 5fps at 500k, it means I can probably get 30+ at 100k. If my game normally starts with 1k units, it means I won't have to worry that it becomes unplayable in the late game when the player builds much bigger armies.

If it can help, here are pictures of the game with a grid showing the collision cells. Blue cells are occupied by one or more entities. Some stats are in the green area of the window. The framerate is 60 because the game is paused when there is no physics calculation happening. It drops about half of that when unpaused.

You can increase the number of threads and increase the minimum hardware?

Can you link to a youtube video that show's the kind of thing you are trying to achieve?

21 minutes ago, Mk_ said:

Can you link to a youtube video that show's the kind of thing you are trying to achieve?

I'm trying to simulate physical collision like in total war for example, but it must be fast enough to support 100k units. 

What you want to do is beyond what any games currently do. The Total War games work with far fewer units and they almost certainly perform hacks to simplify their logic.

I will add this: your grid is too small to give you much benefit and I expect the cache coherence is almost zero with rows that long. A more complex spatial database - perhaps even just a large grid of smaller grids - may give you significantly faster lookups much of the time, if it means that the 8 grid squares you're examining are almost always in the same cache line. But what you gain there, you might lose on having the 2 stage lookup, or on having to occasionally search across multiple large grid squares.

I'm assuming your end game is some sort of gameplay like this 

or total war.

Btw, that video, though it's Unity specific, does talk alot about how one should set up their data structures for maximum throughput.  Though they only manage about 40k units at 60-70fps.

3 hours ago, ferrous said:

I'm assuming your end game is some sort of gameplay like this 

Somewhat. I want the epic scale, but with physical rigidity.

9 hours ago, Kylotan said:

I expect the cache coherence is almost zero with rows that long. A more complex spatial database - perhaps even just a large grid of smaller grids - may give you significantly faster lookups much of the time

Indeed. I hadn't thought about it before, but since 8 cells are on 3 different rows, that means there's 3 cache line miss per collision detection. But what if instead of using a more complex system, I would just dumb down the collision detection accuracy? If I let enemy entities walk into a same cell, then all I have to scan is my own cell to see if there's any enemy up close. I'm not sure if that would still look anything like total war though.

9 hours ago, Kylotan said:

your grid is too small to give you much benefit

Why is that?

What do you mean by 'physical rigidity' ?