Just blame it on my poor comprehension skills and over-literalness, but I'm afraid I do not know exactly what part of the magic cycle you are wanting input into; the acquisition of the spells, the advancement of said spells, or the prevention of potential spell usage by the player. all were mentioned, but you rather emphasized one.
If prevention, I.E., how to keep weak players from using strong spells in an organic, less arbitrary way:
If players are drawing energy from the environment into themselves to cast spells, then that is a lot of energy being contained in a rather frail human body. If the spell costs, say, 500 manna, but your character has a manna cap of say 350, than that player shouldn't be able to hold it inside for very long, and, upon releasing the spell, damage would be done to her body. Also, the effort of holding in too much energy should cause her focus to waver somewhat, causing a decrease in either potency or accuracy. Or, you could have it that the chance of successfully casting a spell is ratio of player level over spell level, or some other such equation.
As for Advancement, I.E., increasing the effectiveness of spells as the player levels up/progresses through the world:
I've always been partial to small bonuses that cumulatively increase as one gets better at the skill in question. For example, all fire magic might have an innate, albeit weak, armor piercing capability, say 2%. As the player gets better at fire magic, that small armor piercing will increase, to 3%, then 4%, 5%, etc., eventually getting more effective, especially if you have other such tiny perks, say a burn status effect which will also increase.
As for actually getting better at the magic, you could have it increase with usage, and with finding books that advance that specific magic.
I absolutely adore this idea, but I feel that it really depends on whether your game is single-player or online, and, if single-player, how linear the game is. I think it would lack its appeal if you get these items from bosses that you have to fight to advance the game, because then it would just be a tedious complication of gaining power from killing bosses when you you usually do that anyways, through a boss having a 100% chance of dropping a strong sword, or was guarding the morphball, etc.
As for actually acquiring the spells in the first place, again it rather depends on whether your game is online, linear single-player, or more open-world single-player.
In Castlevania: Symphony of the Night, there are a couple of magic spells (though to be fair, they were not all that useful, and were not the focus of combat), which you could perform. You either discovered them by accidentally doing the requisite movements to cast them, or by buying a scroll which told you how to perform them. Another example of such a system was Legend of Legaia. In it, your attacks were based on a small queue in which you placed in any order your actions(in this case, high-low punches and kicks I believe). If the order and combination of actions conformed to a specific special move, then your character performed that move instead of the actually moves you placed. Depending on how the player actually casts spells in your game, you might do something similar. If the player casts spells by combining power words, than the player might accidentally discover a new spell by experimenting with the order or composition of the words, or by finding books that have a spell already mapped out for them.