I would do the same as your last suggestion. You can make it open source and sell support for it or paid custom features , This way you can also force people by license (like LGPL) to submit patches and fixes and the engine will grow and improve over time.

On the other hand, opening source would require additional work to streamline things, refactor some others and make it easy to use (even if you sell support).

If it kicks ass and is open source, preferably with a permissive license, I would see no reason why people would not get excited. However it may not happen overnight.

In either case, open source or not, building the website and community side can be a huge and thankless drain on your time/energy, so factor that in. In the case of my own engine (Urho3D) I deliberately skipped those to save my time and sanity, therefore there's only github/Google project pages and a Google discussion group up. As a consequence the engine took a long time to get known and still isn't very much known But like said, this was a conscious choice.

Really, the question is this: What does your engine offer that's so great it can overcome the momentum of Unity and other engines? You can't be "as good" as Unity and expect to take any of their share, you need to be better. In fact, substantially better, not just a little better. What you have may be very good, it may be your best work; I don't know your pedigree and experience so I can't really speak to what that means. But it does seem unlikely on its face that any one person could develop an engine--together with the associated tools and workflows--to compete with something like Unity. Oh? You don't have really polished tools and workflows? You're dead in the water.

Now, better can mean a lot of things -- it doesn't have to mean higher tech, or I-can-do-everything-unity-does-and-more. It might mean your engine is more limited but also simpler in a way that people can really grasp. It might mean your engine is the very best engine around for 2.5d, physics-based farming sims. Both niche markets. Unity is very general, and while it may not be tuned to particularly excel in any one kind of game, it seems to do well enough in the broad sense. That's actually an incredibly hard thing to compete with, and as a business strategy it might be better to identify where Unity and other engines are lacking, and to really attack that area to build a following among niche markets where Unity and other engines aren't currently a good fit.

Finally, keep in mind that rolling out a successful business isn't something that happens overnight. Having a product to sell isn't a business. You need a solid business plan, the marketing acumen, the logistics (support, payment processing, legalities, etc) in place, a roadmap for your product, and most importantly you need someone to sell your product to. You don't even have the skeleton of a business until you've got all that sorted.

I would agree with others, it has to be compared to other established Open Source engines (OGRE, etc) and freely available closed source offerings like Unity - the reality is that even if offered for free, you would get a hard time getting people to use it, let alone paying for it (not a judgment on your work, just noting that the existing free alternatives have had a lot more people working on them, for longer, and also have proven themselves with lots of games being written using them).

Also note that there's a difference between an engine for your own stuff, and a supported 3rd party library - I mean yes, if your game is really good, people may take the engine and use it (if it's Open Source), but creating a more general game engine like OGRE needs a lot more work on things like tools, documentation, examples, support, as well as making sure that the API is broad enough to support a range of game types, rather than just the one you are making.

The easier route may be to just use it for your own stuff, and do it for the fun of it - but I think releasing things as Open Source is a good thing anyway.

All good points. I am using a free and open source game engine called Maratis3D. It was made by one person, Anael Seghezzi. It has been in development for a number of years and although it is a very good engine, it has a lot of things to work out.

One main thing I see is that not many people know about the engine. I actually stumbled across it while watching a Unity tutorial on youtube.

I personally don't like Unity. So I think there is room for another engine. I think Maratis has a good shot, but Anael needs more help to implement basic needed features. It relies heavily on Blender 3D which most people in the industry prefer not to use.

He still needs to find a way to make money in some way, so I suggested a content store. Supposedly that wouldn't be a good idea at this stage since even making content doesn't have a smooth workflow.

One thing I have seen is that Anael doesn't put anything in the engine unless it is 100% sure. That is why the engine is so solid and fast.

There are so many people trying to make a buck on engines that it is almost a relief to have a good engine that is free and open source for beginners who just want to make something.

My idea for a game engine (which I am more than likely never to make) I think would be the best type of game engine, but the scale of it is sort of big. So for now I am using this free engine to test my ideas of an engine.