Making a successful entry to the game engine market is HARD, especially now that Unity has such a monopoly on indie mind share. This is true even if you provide source and license for free. Two recent and relevant case studies:
1. Havok released their professional-quality engine (Vision) and libraries with much of their middleware as Project: Anarchy under a free license with substantial source code.
2. Torque 3D, which used to have a paid license model, is now completely free and open source with a permissive MIT license.
Even these free, commercial engines have had a hard time gaining traction against Unity, however. More to the point, with professional quality engines with at least some track record in commercial games now available free, you'll have a hard time attracting users if you charge. It's absolutely a huge downside to not get paid for hours of work...but maybe you should try to develop and release a game using your engine. This would help build credibility for your engine, and it's easier to sell a game than an engine.