Since you said you'll be very specific on your game engine, I think I don't mind for you to continue on. Of course, feature creep may get in your way so be careful for this, time will tell. I just hope it's just a hobby.

I think you should create a little risk management, but of course that depends on your goal. For example, if all you really want is the engine, you'll actually don't mind to give your life to make your specific 3D game engine. However, if your goal is the game, you'll set about X years phase for the game engine. If it's not quite feature complete/game development ready, you might want to switch to use current available game engines with extra 6 months of learning phase, or if the game engine is usable even at 50 - 75% completion, you can finally start the game development.

This way, since you've been dealing with the engine for a few years, fail or not, you know the scope of what specific graphics features are, and using someone's engine would be easier cause you know what you're looking for and you know how it works. So it's a win-win for you. Just set a deadline to motivate yourself on how long does the engine should be completed.

Trust me, no matter how many years/features you put on your game engine to be perfect, when you finally start creating the game using that engine, you'll always want to fix your engine where someone else could do the job. I've been there, and it's quite a long journey for one man's brain to go back and forth from engine to the game itself just to fix a little something (happens a lot and not to mention architectural/design failures; it's crazy, you might want to take a little vacation when that happen. which usually you'll end up letting it happen until the next big refactoring, the engine v2 roadmap).

Forgot: About Java/C++, since you want performance you might want to try C++. About the graphics API, that depends on your target platform.