|I'm a firm believer in the maxim of only coding what you need to. You can plan for the future and develop libraries and components but you tend to end up only ever completing those libraries and components and never the products they're supposed to be used in. So I find it more fruitful to just code things as I go along and refactor them into libraries, components, modules, classes etc as necessary. One of my old university lecturers said to us, "it's not reusable until it's usable", which is quite wise. In other words, only code what you need; that guarantees that it's useful and is a candidate for re-use.|
As a result, I don't write engines. (Quick prototypes and the like excluded.) I write very basic game loops and factor out the subsystems as needed. I start off with a game and finish with a larger game, without ever just having a technology demo or engine.
It is actually possible (and quite feasible) to combine both the "only coding what you need to" approach and the "building an engine/framework/infrastructure" approach. The catch is that you can't combine them in the same system for the same project. It takes multiple projects, and the experience of those multiple projects, to allow this to happen effectively.
I'm perfectly content to call myself an "engine guy" at this point, and most of my day-to-day work involves creating the core components/services that the other engineers rely upon for doing the higher level logic for the game simulation, user interface, and so forth. When considered by itself, that kind of foundation appears as if it was created in a vacuum (as "technology for technology's sake", like many prospective engine writers tend to make). But when you look at these systems in the context of past projects I've worked on, you'd see that the vast majority of them are either:
A) things that I tried on a past project that worked really well,
B) things that started out differently on a past project but which gradually turned into this in the end (because it worked really well), or
C) things that i didn't do on a past project but by the end I really really wish I had because I had a concrete need for them but there was no time/room/whatever.
Basically what this means is that you should "only code what you need", but with an understanding that sometimes "what you need" should be part of your stable foundation, and it usually takes several projects to figure out exactly what should go where.
One thing is for certain: if you're doing an experiment (or any completely new system you don't have sufficient practical familiarity with), don't make it a core dependency like this. Keep that stuff as leaf code so that you keep the dependencies low and preserve your ability to iterate the system's design. The more you iterate, the more you see what the system should converge on and how it should be more appropriately arranged, and you can take advantage of those changes the next time around. These don't even have to be completed projects (although the more completed they are, the more complete your experience with them will be, of course). As a product of this experience, you will gradually get faster and faster at doing this kind of refactoring and reorganizing, and you'll get better at designing more robust and usable systems in the first place.
Keep fighting the good fight, and keep shooting for perfection. Just understand that perfection is a limit that you try and converge upon, and not a goal that you can actually achieve. Every project will have its failings in this regard; what matters is that you learn from those failings and reduce them the next time around. Don't kick yourself for making mistakes, because those mistakes may be some of the best lessons you'll ever have.
And BTW, if you're looking at getting into the industry, and you find that you're spending more of your time on core systems than gameplay code (preventing you from getting a game done)... that's fine! Just take that focus and combine with some people who do
focus on gameplay code, and who are willing to use the core systems you create, for a better end result. That's what real teams do, after all. You don't necessarily need (or want) several different people all wanting to make the best wizbang graphics engine, or the best scripting system, or whatever... but if you can get people who want to handle different tasks (in different layers of the codebase), then you can start to work out dependencies between those tasks and what needs to get done when in order to keep everyone productive, and so forth. Those kinds of lessons and experiences are just as valuable as the programming experiences themselves, if not more so.
Oh, and evolutional: thanks for the compliment on COTC; I'm glad to hear some folks are still getting value out of it. :)