You are misunderstanding the term "open source". (As almost everybody does nowadays, eh...) It does not mean that you need to give away the whole thing at no charge. You can release the source code to your engine, but keep the assets proprietary. Let me say it again - It is possible to keep the art assets proprietary and still sell the game, even though the code is classified as 100% free/open source as defined by the most diehard advocates of it.
Free/open source software is often criticized that it's hard to monetize, as once you sell the software to someone that someone can give it away for free - well, the situation with games is different. If you have a gigabyte of proprietary art assets then even though people will be able to give away your game engine for free they won't be able to give away the whole game. (Hence you'll still get paid.)