IANAL (I am not a laywer)
First, all code/art/music/whatever you create is automaticly copyrighted. If someone steals it, all you have to do is prove that you posted it online before they did (simplification. All you have to do is prove to a court that you created it). The GitHub page makes it public, and records the date.
Second, licensing is you basically saying, "I have these rights, but I'm willing to share some of the rights with you, if you follow certain conditions..."
Licensing doesn't have to be open to everyone, it can be just specific people. Additionally, you can license the code with one set of conditions to one group of people, and a completely different set of conditions to a different group of people.
This means, to satisfy your goals, you can (and should) have a different license for commercial vs non-commercial use.
BSD wouldn't be good (to meet your personal goals), because it basically says that anyone can use it even for commercial work.
LGPL also wouldn't be good for the same reasons.
GPL is a better option for non-commercial work, but even GPL allows commercial usage. The thing with GPL, however, is it forces anyone using it to release their source code of their project under the GPL as well, and most commercial software doesn't want to use it for that reason.
You could just have a plain-english agreement for the non-commercial version, and then negotiate for the commercial licensing. It'd have less legal strength then a lawyer-written agreement, but it'll still have a decent amount of strength.
That said... What is the likelihood that you'll actually make any money licensing your engine? Does it offer features that Unity or the open-source Torque don't have? How does it distinguish itself against the hundreds of competitors? If the answer is, "It doesn't", then you should consider just making the entire thing LGPL.
LGPL means that if someone modifies the engine, they have to share the modifications. LGPL means, yes, they can use it for commercial games... but if you aren't going to make any money from licensing, then why not let people get real use out of it? I fully get you wanting to profit off your hard labor (as do we all!), but if the choice is that either they won't use it, or they'll use it for free, then I would suggest letting them use it for free.
My suggestion is, if you yourself don't think you'll make much money on it, offer it fully for free, even for commercial works.
If you do think your engine stands apart from the crowd, then yes, license the commercial version. But if it really stands out, then you'll really want to hire a real lawyer as soon as possible.
If you don't think that you'll make enough money to cover the lawyer's fees, then it probably isn't worth it to try to charge for it. But this is my opinion. You know your work, you know it's potential compared to engines like CrystalSpace and Panda3D, and you know whether it's worth the effort involved in trying to sell it.
Even if you commercially license it, and someone steals it, you'll have to pay for a lawyer to sue them anyway.
Edited by Servant of the Lord, 25 June 2013 - 06:15 PM.