There are a lot of topics with this title, and almost all of them repeat the same names: Unity / GameMaker / Torque3D. I can add Godot, which is pretty new, for the sake of the post. You didn't specify 2d or 3d neither, even if something tell me 3d.
The game is 2d top-down(bird eye) AI robot battles like robocode.net
Is box2d scalable to 100000 characters? The problem with the engines out there including box2d is that they run a loop and in each loop check for collisions between objects. but if instead of checking for collision on each object in each loop if we calculate when the next collision for an object occurs using equations of motion then i think it will be the most efficient.
This is more a hardware problem. My on-broad Intel HD Graphics 3000 can render around 8K 32*32 sprites with same texture at 60fps, while doing game logic and rendering simple background. With some decent graphic card you could do maybe like 50K simple sprites or particles, but 100K "characters" don't sound promising.
This is meant to be run on server, while showing only small part to frontend.