I should have just made games and derived an engine.

The two are mutually exclusive in my opinion. You either write what you need to make a specific game work or you write an engine that allows you to make several games once that engine is complete. Otherwise you'll quickly loose focus and never finish what you started.

To answer OP's question, I started writing an engine 5 years ago and I haven't finished yet. If you want to write a game then I'd suggest pick an already complete engine that suits you and focus only on creating a game. If you want to write an engine then go for it! It's fun, you learn lots, but just be aware it can take a long time.

If you've worked 5 years on an engine without making any games with it you might have a big problem though, you got 5 years worth of work that hasn't been properly tested. Making complete games using an engine is the only sane way to test its design, performance, stability and workflow(a big problem with many hobbyist engines is that they are virtually unusable for large projects, despite shiny tech demos) issues tend to pop up quite quickly when you try to use pretty much any engine(even good engines have issues, allthough they tend to be less severe) for a real game and you want to catch the major issues as early as possible.

This is true, and it's true about any product that is created, not just software. However if you have a background in what it is you are creating, set out with some clear design goals, and use an iterative design through regular testing then problems will more likely be caught early too.

But yes, the best way to test any product is to use it in a real life situation.

A generic engine is too nebulous a concept, it can be so many things, that in isolation, it will never be done. There's always more things it could be doing, or better ways in which it could be doing the things it already does.

IMHO, an engine needs to be driven by a customer. ID Tech is always driven by a new Quake/Doom/etc, Unreal Tech is always driven by a new Unreal/Gears/etc, CryEngine is driven by a new FarCry/Crysis, an so on.

Gamebryo failed because they never ate their own dog food (they made/sold a crappy engine, without ever making their own games on it).

The "dog food test" should always be applied to an engine, so I'd avoid others like BigWorld, etc, as they failed to make a game on their own tech...

You have to eat your own dog food.

Pick a game you want to make, figure out what feature that game needs from an engine, and then develop your engine to support those features!

Also, the only way that you can test whether you've done a good job is to actually make the game! So, you should always make the game in parallel with the engine. Don't finish the engine first and then make the game, because that just wont work. You will find problems with the engine once you try to use it, and you will have to re-do engine work while making the game, so make them side-by-side, so that they can both influence each other.

Then if you want to keep making the engine, make another game using the same process!

The two are mutually exclusive in my opinion. You either write what you need to make a specific game work or you write an engine that allows you to make several games once that engine is complete. Otherwise you'll quickly loose focus and never finish what you started.
I started writing an engine 5 years ago and I haven't finished yet.

I think you just disproved your own point. You're making an engine that has no specific finish line, no goal, no hard list of requirements to fulfil, and no way of evaluating/testing it's merit... so you've not finished

Having an explicit purpose for the engine (it must support the needs of this game) gives it a solid focus, and a framework for evaluation.

I think you just disproved your own point. You're making an engine that has no specific finish line, no goal, no hard list of requirements to fulfil, and no way of evaluating/testing it's merit... so you've not finished
Having an explicit purpose for the engine (it must support the needs of this game) gives it a solid focus, and a framework for evaluation.

Maybe I was far to broad in my statement. When I started, I had specific design goals from previous experiences, an end goal, and time frames. Of course I didn't always meet deadlines but I did my best to meet them. I set sprints on work and goals to achieve and that kept my motivation high. I did achieve what I originally set out to achieve and I am happy with with my efforts. By not finished I meant things like bug fixes, optimisations, and "nice to have" features. I had purpose when creating it, and it was being designed side by side a game, a number of games over time actually.

What I found difficult being the only coder was the artists needed to see the game coming together to keep motivated (which is totally reasonable and I understand that). If I didn't stay on top of the game then it would hinder their workflow of iterating over the artwork and animation to improve on it. The issue there with me was the game was dependent on me getting some feature in the engine working, not necessarily finished, just working. So it became a situation of constant context shifts for me as to not slow the artists down too much.

In my current position we have an engine team, and product teams. The product teams can focus on making the products that are customer driven, without worrying too much on how the underlying tech works. While the engine team can of course just focus on the core tech. If the customers want some new tech then that obviously falls on the engine guys. In this separation it is rare for a team to slow another down.

So the point I was originally making was it can be hard as a lone coder to create a game and reusable engine side by side.

I agree with everything that has been said about game engines failing because they have not passed the "dog food test". I've used gamebryo and it was a horrible mess.