Instances are the devil. It sounds like you want to play some sort of coop or something. The goal of my game specifically is to be sandbox off rails with actually massive player interaction. I am sacrificing graphics for gameplay in order to reduce computer loads as opposed to minimizing the number of moving objects like creatures and players.
In any case creatures will be a little more intelligent than just charging in. But creating creatures that can fight twitch style with players? Thats just not realistic. They might have simple patterns like range attacking from the farthest possible distance or small fast creatures trying to run if they get too damaged. To be honest though that can create a lot of problems. For instance loot and exp wise, if you are fighting multiple mobs and a lot of them get away, whats the gameplay there? That's why designers tend not to make creatures run. All it means is you waste time and lose rewards. Sure you could make a tracking thing, but what % of people really want to do that?
Use your imagination. Incorporate these behaviours to be more than just being harder for players. Also if something is harder, then you can increase the rewards.
Your evaluation with trouble with loot and xp is based on the "old world" where you advance by killing near stationary evenly distributed creatures.
Use these behaviours to make some targets hard to get. To succesfully take them out, you need some special tactic, often in cooperation with other players. The potential rewards would be so that players want to take these out. There will still be easy targets, but they won't offer much of a reward.
Preventing high reward targets from escaping would be a big part of succesfully getting rewards, while tracking them down would be an optional plan B. They could be easier to catch after chasing them away once, but you waste time. The reward could also be something else than regular foes. You may have to face regular foes on the way though. In Tracking down a rich resource site you may discover that a group of npc's is alreay extracting them. The most powerful foes are in the group that carries resources back to base. Striking when they are gone is a good idea, but if you are discovered, they won't leave.
The problem is that if we are trying to make more realistic creature behavior we have a problem with rpg systems period. For instance in many real world fights you need to get that one shot in and its a kill. RPGs involve incrementally lowering a health point value. If a wolf fought a human in real its all about who gets that critical lethal hit in first. More realistic behavior can't exist in a vacuum. A lot of the "real world" animal behavior is all about things that work like real world fights. There is none of this healing shit that happens in RPGs. Darting in for a hit, running back, wearing people down, it just doesn't work in a game where the corresponding results of the actions don't exist. Normally realism wouldn't be our primary consideration in designing a game but the point of changing mob behavior would be to make it more animal like, in other words more real, so in this case objection from reality is viable. Unless creatures have healing and ranged power to compete with the kind that the players have it is in their best interests to stay right by you and keep hitting you if they want to win.
In regards to just running away when too badly damaged, there are limited ways we can deal with that from a player perspective. Given enough health as long as you can't box the creature in it can escape and how are you going to follow it? Would a fleeing creature leave clues? What clues? Would it be the same for each creature? Would their just be a tracking skill? How would it level? How would it decide if you lost the creature? Its just not realistic to code that.