A couple of options to look at:
UDK. I don't personally know as much about this one, but it's definitely capable of what you're after and again you can start development for free; it's then just $99 when you want to publish, and royalty payments if you make over $50,000. You should be able to find out more on the website and in the faq.
Leadwerks. Again, well supported and capable of what you want to do. Programming in C++ or Lua, and again there's some form of visual editor offered.
I remember someone suggesting a pretty capable looking 3d engine that didn't require any programming recently, but unfortunately I can't for the life of me remember what it was or where the recommendation was. Hopefully someone else might know what it was!
All of these can be tried for free, so I'd suggest checking out any of them that seem interesting. I'm sure others will have some good alternative suggestions and further feedback on these ones as well.
For reference, you can achieve the "less real" look you're after with the right artwork even if the engine is capable of "better".
Hope that helps give you a starting point.