I do not know too much of these open-source game engines, but from what I see in terms of rendering quality they are not that great.
Ogre3D is awesome. It is a very good rendering engine. Check out the demo on the website I linked. Torque 3D is also very good looking(especially with advanced lighting on).
In order to be able to modify it to do what you want, you still need an understanding on how the basic stuff work under the hood, right? So, is it better for me to write small games without an existing game engine just to learn the necessary skills to use a better one in the future? Or, I could just try and use it without doing that?
Yes you need to know 3D math and be good at programming. You should start learning to program simple games first and then move to an Open Source engine, open up the hood read the code, play with the scripting language and level editors or what it now offers etc. etc. It is a long process even with an Open Source engine. But the journey is also fun.
I didn't thought on building a game engine, at least not a multipurpose, reusable tool that everyone of them tries to be. I just want to build a game without a game engine.
Also read this: http://scientificninja.com/blog/write-games-not-engines
Then it should make more sense
Edit: no comments below the article anymore as far as I can see