I recently started programming a 2D RPG and my goal is to have an open world with hundreds of NPCs, very much like in TES games like Skyrim. The idea for the world is to be a classic 2D tile grid, about 2000 x 2000 tiles, and the NPCs should behave acoording to AI scripts that each of them will have assigned - I basically imagine each NPC having an AI script for different events, like for example "being attacked" or "ceratin amount of time has passed" etc, on which the script will be run. I naturally came to a problem of how to implement the game engine that could simulate the whole game world in real-time.
Problem 1: How do I design the engine to be able to run the AI scripts of all the hundreds of NPCs in the game at once? I know that some open world games, like GTA, only update the NPCs that are in the player's close area. However I cannot do this, I need my world to work independently on where the player is, the events need to be happening even when the player isn't around, like in Skyrim. So I need to run every NPCs AI script plus I need to compute all the interactions of the NPCs with their environment, such as detecting colissions, have them fighting each other etc. But I have a feeling that this would be very bad for performance - I would like to know how games like Skyrim deal with this. My guess is that they use real-time detailed simulation for NPCs that are in the player's close area and they handle all other NPCs in the world only once in a while, i.e. once every nth "game frames", plus they may use some simplifications of the "NPC - environment" interactions (like teleporting NPCs instantly instead of having them really travel step by step with colission detections etc.). These are only my guesses and I don't really know how to make the simplifications, or whether I can use them. I would love to know how this is actually done.
Problem 2: Do I need to have the whole world plus all the interior instances loaded in memory at once? I would like to only load one small part of the world where the player currently is in order to save memory, as having 2000 x 2000 tiles plus all the NPCs, objects and items loaded can take a lot of space. However I cannot do this, because, as I mentioned above, I need to be simulating the whole world constantly, so I am forced to have it all loaded in memory. Again, I would like to know how Skyrim does it, because it obviously cannot be all loaded at one time, at least not with all the details.
I've been searching for answers to these questions for a few days now and I cannot find them, so I would be very glad if any game developer or anyone who might know the solutions would give me at least the basic ideas of how these problems could be dealt with. Thank you very much :)
Technical details: I'm currently using Python with Pygame and Numpy, but I could switch to C++ later if performance shows to be an issue.