Just blame it on my poor comprehension skills and over-literalness, but I'm afraid I do not know exactly what part of the magic cycle you are wanting input into; the acquisition of the spells, the advancement of said spells, or the prevention of potential spell usage by the player. all were mentioned, but you rather emphasized one.

If prevention, I.E., how to keep weak players from using strong spells in an organic, less arbitrary way:

If players are drawing energy from the environment into themselves to cast spells, then that is a lot of energy being contained in a rather frail human body. If the spell costs, say, 500 manna, but your character has a manna cap of say 350, than that player shouldn't be able to hold it inside for very long, and, upon releasing the spell, damage would be done to her body. Also, the effort of holding in too much energy should cause her focus to waver somewhat, causing a decrease in either potency or accuracy. Or, you could have it that the chance of successfully casting a spell is ratio of player level over spell level, or some other such equation.

As for Advancement, I.E., increasing the effectiveness of spells as the player levels up/progresses through the world:

I've always been partial to small bonuses that cumulatively increase as one gets better at the skill in question. For example, all fire magic might have an innate, albeit weak, armor piercing capability, say 2%. As the player gets better at fire magic, that small armor piercing will increase, to 3%, then 4%, 5%, etc., eventually getting more effective, especially if you have other such tiny perks, say a burn status effect which will also increase.

As for actually getting better at the magic, you could have it increase with usage, and with finding books that advance that specific magic.

a fairly basic idea that might be interesting food for thought could be a means of using the loot from certain enemies as a means to 'transmute' (to some degree) reagants that would be forever with the player from that point.

I absolutely adore this idea, but I feel that it really depends on whether your game is single-player or online, and, if single-player, how linear the game is. I think it would lack its appeal if you get these items from bosses that you have to fight to advance the game, because then it would just be a tedious complication of gaining power from killing bosses when you you usually do that anyways, through a boss having a 100% chance of dropping a strong sword, or was guarding the morphball, etc.

As for actually acquiring the spells in the first place, again it rather depends on whether your game is online, linear single-player, or more open-world single-player.

In Castlevania: Symphony of the Night, there are a couple of magic spells (though to be fair, they were not all that useful, and were not the focus of combat), which you could perform. You either discovered them by accidentally doing the requisite movements to cast them, or by buying a scroll which told you how to perform them. Another example of such a system was Legend of Legaia. In it, your attacks were based on a small queue in which you placed in any order your actions(in this case, high-low punches and kicks I believe). If the order and combination of actions conformed to a specific special move, then your character performed that move instead of the actually moves you placed. Depending on how the player actually casts spells in your game, you might do something similar. If the player casts spells by combining power words, than the player might accidentally discover a new spell by experimenting with the order or composition of the words, or by finding books that have a spell already mapped out for them.

Another idea might be that they have to learn fire from say, studying a flame and learning the properties of it. That way it forces the player to travel to obtain new skills, and you can keep the higher level stuff in areas that are harder to reach.

What would studying entail? I feel like that is the key to making the process more interesting than simply "go to place x and pick up flame y and get skill z."

The actual study would be up to you. It could be as simple as having the player simply wait while the character learns it, to having a mini-game(simple or complex) that can either take the scientific route of putting together atoms or putting together elements, or items, or even just having them actually make fire by somehow making a spark with given items, and getting that spark onto something that will keep flame. Maybe even figuring out what takes the flame best in what situations.

If you can find a copy of it, Middle Earth Role Playing (a tabletop RPG from the 70's/early 80's) had a magic system similar to what you are describing. Every spell was a 'list' of ten effects. The more powerful a mage you were, the further up the 'list' you could cast. There were lists for elements, defense, all sorts of spells.

I'm not a big fan of spending real time 'training'. I figure my character does that off-screen somewhere. Charging money to progress may be mundane, but it serves a purpose as a money sink. A lot of games suffer from going from not enough money in the beginning to excessive amounts after the mid-game (Vtm:Bloodlines I think I could have wallpapered my haven in hundred dollar bills if I wanted...).

'Experimenting' sounds great in theory, but remember there will be a FAQ up within 48 hours that lists every possible combination and what it yields. You could go with random results from 'experimental' reagants or something (the 'seasoning' to the base recipes, if you will), but that brings its own set of headaches to the table.

I'm developing an RPG system which uses something similar - a certain kind of magic user can draw energy from his surroundings and other sources and use them + own knowledge to work a spell. My suggestion comes directly from my system.

In yours, you describe "type" and "target" spells. However, what happens if you allow a mage (as they gain power) to apply multiple "type" levels to each casting? So, a level one mage (or skill 1, whatever system you use) has one Type slot and one Target slot. But when that mage gains in power, they earn a second Type Slot. They still must pick a single target (unless you can think of a good way to utilize multiple target slots...) but they can increase the power of a spell by using multiples of the same Type (2 Fires + 1 Other == Bigger Fireball!) or even get more complicated spells by mixing Types (1 Fire + 1 Air + 1 Other == Fire Tornado?)

It has the possibility of getting very complex, but with a few ground rules on how to handle certain uncharted multiples you can handle much of that. For instance, you could have a "general energy" Type, and any time a mage tries to cast a spell which isn't actually it's own spell the extraneous Types get turned into general Energy. So, if 1 Fire + 1 Water + 1 Other wasn't actually a spell, it would turn into the spell 1 Fire + 1 Energy + 1 Other. Thus, you draw everything back down into a baseline which you make sure to define behavior for.

My only question for this system as you've defined it (with Target being a special magical sphere of its own) is how many targets you're really going to have? In my system, the target is not a magical effect in itself, and mage level basically directly corresponds to how many Spheres he can throw into his spells.