Still, it's a very cool idea and its an interesting approach to preventing grinding on low level creatures...though just diminishing XP rewards on lower level mobs accomplishes this as well, and is the usual route taken.
A few years ago I was thinking about the diminishing returns from mobs - and similar to what you said, I thought that people do learn when faced with new and unknown things.
I considered a system where the amount of xp you get from a mob is inversely proprotional to the number of mobs of that type you've fought. So, the more you fight the same mob, the less xp you'll get out of it. On the flipside, the more you fight a certain mob, the more damage you'll deal. (both an determined by an asymptotic function, the xp's asymptote being 0, the damage multiplier being something like 2 or so)
Basically the idea is that if you fight something for the first time, you learn more from it, and the more you fight it the less you get out of it - but the better you are at fighting it.
Based on what you said, this can be extended to humanoids, by assigning each humanoid npc some sort of fighting style - which would be the determining factor for xp. If you fight npcs with the same fighting style, you gain progressively less xp from them. The damage can be a combination of how many overall you've fought + how many from this fighting style you've fought. (basically you get better at reading humans, and better at countering the fighting style)
typed that up in a real rush hope it makes sense.