Usually, when a character "fights" an excessively high level monster the character dies or runs away, without earning experience or any other reward. Your formulas for experience points and levels only need to behave well in a very limited range of monster levels above or below the intended enemy strength.

There are various approaches to calibrate how many experience points beating a monster should be worth. For example, we could reduce the gain depending on character experience. If the character has C experience points and the monster has a reference value of M experience points which simply reflects its power, victory should give the player f(C,M) experience points, with some obvious constraints:

• C>=0 and M>=0
• f(C,M) >=0: the character never loses experience.
• f(C,0) =0 (approximately): monsters of negligible strength are worth nothing
• f(0,M)=M (approximately): an arbitrary scaling to connect the scale of C values corresponding to character levels and the how many victories are required to earn them
• f(a,M)>f(b,M) if a<b: less experience at higher levels for the same victory
• f(C,a)>f(C,b) if a<b: more experience for bigger monsters, at any level
• f(C,M)=0 if C>=M (approximately): fighting reltively weak monsters should be useless

Given these constraints, an obvious simple formula is f(C,M)=M*M/(M+C), which equals M/2 if C=M.

It can be generalized to f(C,M)=M*(M/(M+C))^k, with the exponent k greater than 1, to obtain a sharper "knee" in the curve the larger k is.

I suggest looking into how the Borderlands series handles leveling up.

(This is what I have noticed from my playthroughs)

The higher the enemies level is, The more xp you get upto a limit.

So killing a enemy that is around the same level or lower will get you like 1xp, But 5 levels higher gets you ~100-~300xp.

Also the harder the enemy is (Badass, Super BA, Ultimate BA) applies some kinda multiplier to the xp.

HTH