Just to point out, i never said vanician magic (which seems to be you prepare and choose spells beforehand). I might just limit the number of uses per dungeon for some spells/skills.

For myself, I'm not suggesting Vancian magic, nor saying that it was one of the options that you listed. Rather, I think that limiting spells and skills might incur one of the problems that can come with Vancian magic: the "Too Awesome To Use" issue that I mentioned previously: The player, knowing that they'll lose a given spell for the rest of the dungeon if they use it, might end up never using it at all; conversely, the might use it and then end up getting killed by its lack in a fight against a foe that they didn't know that they'd be fighting. (I don't mean to suggest that those are the only two options--I'm just listing them as cases that concern me.)

I like Deflinek's suggestion above: Limit spells for a given fight (calling thus for interesting decisions on the player's part), but reset those limits after each fight (allowing players to feel that they can use their spells and skills in combat).

The problem with mana only is that it might end up using some classes just for one thing, for example if a cleric can heal and do some damage skills (which both cost mana) you might end up ONLY using the heal and not wanting to "waste" mana on a more diverse set of skills.

So you're going to introduce a character who is specialized in healing, make his healing skills way more powerfull then his other abilities,

then complain that players use his best skill instead of randomly choosing a skill ...

Yes. If a single skill is always the best option, then players aren't really making a choice. They're simply selecting the obvious option from the list every time over and over again until they get victory. That's not much of a game.

Good game design provides interesting challenging choices, where you try to figure out which ability is the best for that moment, and have multiple factors to consider. This can still be done with mana and a diverse set of skills, but you do take the risk of making those extra skills pointless if one skill is clearly better than the rest.

Yes. If a single skill is always the best option, then players aren't really making a choice. They're simply selecting the obvious option from the list every time over and over again until they get victory. That's not much of a game.

Make an improved heal-spell with 50% more healing and 100% more mana-cost.

Add the cost-less basic attack.

Now it's situational which spell is best.

Right. Like I said, it can still be done with mana, and that's one way to do it. First, you need to identify potential problems, like making one skill way too powerful to the point that players choose it every time, like you described earlier. If you analyze the problem, you can come up with solutions. The solution could be a balance to individual abilities to work well with mana, or an adjustment to the magic system so it doesn't use mana at all. Whichever you choose, it helps to examine the potential problems of the system.

Why would hard limits be frustrating? The basic attack would be infinite obviously.

Well, it depends on the design of the game, of course, so I'm sure it would be possible to get it to work smoothly. But one problem I first thought of is that hard limits don't reflect the size of the dungeon. Surely some are going to be (much) bigger than others?

But having said that, sure, I guess there's no problem with any system if it's designed right. I'd probably only use one limiting factor for a given class of ability though. My only real experience with playing this kind of thing is XCOM: Long War, where special abilities and psychic powers have cool downs, and special equipment (arc throwers, grenade launchers, etc) don't have a cool down but do have a hard limit.