Hard for me to tell what exactly you want.

One way I've day it is as some randomness to the damage. Like:

int r = rand(0, 1000);

float f = r / 5000.0f;

int damage = (attack * (f+1.0f)) - defense;

That would give up to 20% higher attacks sometimes. Another thing I've done is simply like:

int r = rand(0, 10);

if r == 1 {

miss, critical hit, etc

}

Oh ok, it almost looks like a hermite curve but I'm not great at math. A math person will have to solve this. If you want to try hermite interpolation then use 0, 0, x, max I think as the parameters.

As you can see from the image on wikipedia for hermite spline interpolation (which is really easy and I can give you code for which is like 2 lines) the dark blue line looks kind of like what you want:

My code for this is on GitHub: shim4/src/interp.cpp at main · troutsneeze/shim4 (github.com)

Abstraction please. Your damage formulas are a means to an end (having “fun” combat in your game), but not the only one. An exchange of blows that ends when someone feels sufficiently hurt is a straightforward, understandable baseline; but what do you want from it?

• How long should the fight be? Too few turns, and it becomes too brutal and too short to allow nontrivial tactics; too many turns, and it becomes boring as any tactical approach drowns in waiting and repetition.
• What opponents and weapons should have shorter or longer fights? If “realistic” battle durations fall outside the sweet spot you should probably compensate by avoiding such bad fights altogether, or by stopping earlier or later than usual on the first blood - death spectrum.
• What numerical range of damage and hit points values do you need? What are the weakest and strongest possible attacks, and the weakest and toughest creatures? Do you need headroom for special cases? Do you really want to distinguish similar creatures and attacks (e.g. rat vs. mouse, sabre vs. sword) with quantitatively different ratings?
• How random should the combat be? Do you want the underdog to win once in a while? Generally, random damage increases tactical complexity (more states for the player to deal with become possible, and their probability needs to be estimated) and enables meaningful risk taking instead of just grinding away hit points faster or slower than the other party.
• Why do you like minimum and maximum damage guarantees? Minimum damage will, for example, allow a band of terribly weak but numerous enemies to kill a powerful player character; maximum damage will take away the rewards from a player who tries to stack enough training and items to reliably defeat increasingly powerful monsters with one or two blows.
• What tactical decisions are available for the player? Only two simple ones are implied in your description: increasing attack power or defense power (with different training and equipment) and retreating before seriously risking death. Regardless of how much damage is dealt, more complex rules, with more resources and more complications, seem necessary; traditional tactical dimensions include damage types and different defense effectiveness against them and combat maneuvers beyond a plain attack.

As for concrete formulas, they should be calibrated to interpolate and approximate some well chosen test cases, for example that a fighter with sword, shield and chain mail should defeat a brown bear with better than even odds, with a moderate number of sword hits and modest damage.

Relying on the difference between attack and defense scores makes very low damage situations unavoidable; you could, instead, make attacks random, either a significant amount of damage or a dangerous failure.