Hi, I have absolutely no idea where to put this. I don't think it really belongs in the lounge, but it doesn't really feel like it belongs in programming either (which is almost where I put it).

I've been wanting to make a game/engine for a while, and rather than continue to make excuses to myself like I have for the last 9 months (blah blah, new job, too tired, game just came out, blah blah), I decided to stop pussyfooting around and just do it. However, over this long period of time, I've found that I've spent most of my time thinking about one thing in particular. While not terribly pertinent to the question, that thing has been resource management (asynchrony, caching, hot-loading, and various other features). The amount of detail I've put into the API and design of this one thing got me thinking of a question:

When doing this sort of thing, do you guys usually design/implement an engine depth-first (large, complex subsystems relatively fleshed out before moving on) or "breadth-first" ("minimum viable" subsystems to get a big-picture thing running)? I can see advantages to both. It's not something I'm going to agonize over, since I've learned better than to fret over the small stuff, but it's made me curious. It's not strictly an engine question either I guess, as it really applies to anything that's a loose collection of major systems.

Neither.

I would develop a game and implement the features I need as I go - once the game is finished/I'm bored with it then I'd break out any reusable code, clean it up and repeat the process.

That is how I'd end up with an engine.

What phantom said.

When you are designing anything, whether a "breadth-first" or "depth-first" approach is better depends on the system itself and the context you've already got around that system. Making the choice in a vacuum is pointless. Or at least, both options are equally valid.

The important part is that you should not design and build the engine in a vacuum, because you won't build something that is worthwhile or usable unless you've already been doing this for years and years already (which sounds like is not the case?). Even if you have, though, building an engine up around a game, even a simple one like Asteroids or Space Invaders, can give you the framework you need to focus on which systems (and which parts of those systems) you need to build to get things moving, and which you can stub out or refactor later.

What phantom said.

When you are designing anything, whether a "breadth-first" or "depth-first" approach is better depends on the system itself and the context you've already got around that system. Making the choice in a vacuum is pointless. Or at least, both options are equally valid.

The important part is that you should not design and build the engine in a vacuum, because you won't build something that is worthwhile or usable unless you've already been doing this for years and years already (which sounds like is not the case?). Even if you have, though, building an engine up around a game, even a simple one like Asteroids or Space Invaders, can give you the framework you need to focus on which systems (and which parts of those systems) you need to build to get things moving, and which you can stub out or refactor later.

I know the whole "write games, not engines" thing (and in fact read your blog post on it ages ago), but I somewhat disagree with it applied carte blanche. I feel there are absolutely some things that can be designed independent of the games they run, like asset/memory management, logging, and utility stuff like procedural generation and file caching.

I'm not trying to do something like write a renderer without an anchor. I've tried that before, and come to the conclusion that it's just a stupid approach.

All of the examples above are unique in that they're feasibly completely standalone components. In fact, there are a bajillion libraries that do the latter two, and there's no reason there couldn't be one for the former (aside from the general lack of a need I guess). As opposed to something core like rendering, which kind of has to be done organically based on need for a given game, it's actually realistic to build the aforementioned systems both ways. Like you said, both options are equally valid in these kinds of systems, I was just curious on how people prefer to approach them.

My apologies for not being more explicit.

I feel there are absolutely some things that can be designed independent of the games they run, like asset/memory management, logging, and utility stuff like procedural generation and file caching.

Yes, there are a lot of libraries to do those things. The decent ones were all written by people who had enough experience building such things that they could make the determination for which features to support (or not; an equally hard decision) based on that prior experience. Or they had a project in mind they were building the library along with that helped. I'm not saying you have to build a game, all I'm saying is you need one of those two things (and maybe you have the experience, I certainly don't claim to know) otherwise you'll have issues. I'm saying that looking at depth-first or breadth-first, as in your original post, is not the most effective decision to be spending time on. It's better to spend that time on what the API will do (or not) and what implications that has both on the API surface area and what you need to cart around inside the guts of the API (in effect, that you need to use some arbitrary-but-varying combination of the two approaches you initially proposed).

The reason is because there are a lot of good ways to build all of those things, a lot of solutions that have a fairly balanced set of pros and cons to them. If there were one true solution to any of those problems we'd all be using it (and there'd be no point in your reinventing it), but there isn't and we aren't (so there's potential value in your building another go at it beyond your own edification). A game (or some other concrete project to build in conjunction) helps you decide which particular pros and cons to focus on first for your implementation, so you don't attempt to have all pros with no cons (more or less impossible) or pick randomly and end up with some kind of cognitive dissonance within the tradeoffs you've made. A pure "depth or breadth" first approach to designing such libraries won't really help you solve those more critical decisions, which is why I'm arguing it (in effect) doesn't matter in the most general case.

Plus, if you have something even trivially "real world" to hang your logging API, caching API, asset loading API, et cetera off of, you'll not only have a better frame of reference for deciding what to support/not support, you'll have a more useful context for actually testing them. It's pretty easy to come with an API in a vacuum that looks cool, compact, and self-explanatory and then discover when you try to use it in the real world that the API doesn't account for passing in some certain bit of information that turns out is really useful, or is actually very clunky where you thought it would be elegant, or any number of similar things. While unit and integration tests can help (and are certainly an excellent thing to build), they don't really tend to convey a sense of how somebody would use the system in anger.

I feel there are absolutely some things that can be designed independent of the games they run, like asset/memory management, logging, and utility stuff like procedural generation and file caching.

As a counter point to this, I worked at a place where a bunch of guys of varying experience (some with 5+ years, a few out of uni but very bright) sat in a room for two years and designed an engine, including those parts, without any directing game.

Upon first contact with a game large chunks of it, including things you've listed, were found not to be fit for purpose and were either patched up due to lack of time, replaced if the problems were large enough and simply lived with until a new solution could be found.

While, for example, the concept of the memory pools was ok other things such as sharing and life time management had problems which had to be worked round. Assets where found to not have enough information or where found to be simple 'broken' for the tasks at hand once you got above a simple test level, logging was just about usable, the asset loading/unloading system was found to be broken and had to be worked around (and replaced based on lessons learnt in the next version) and the list probably could go on.

The next version of the engine was much better thanks to this experience and systems which were designed while the game was being made were fit for purpose (and in one case a system, which I designed/wrote, that replaced an old one did the job cleaner, faster and with zero bugs).

There is no substitute for having a game driving your feature development and I've seen first hand the mess which can come about when you lack that focus.

It is also possible to do a semi-approach. You should really always have a game you are working on, but you can always just pull out "generic" code upfront in a seperate engine layer, instead of doing it post-mortem. At least for me this is how my engine was built, I always have a game-project backing it, and this is how I implement new features, but the engine itself is really a seperate entity. I don't say this is the only or the best way, but it can work, and in this save you the need for iterating through a finished project looking for reusable code - which can turn out to be really difficult, at least if you tend to write your high-level game logic code rather messy. Of course you risk having to rewrite lots of stuff if your next game doesn't quite fit the first engines requirenments. But thats totally managable IMHO.

What others have said.

And notice that most of current 'go to' game engine are historically build for games (Unity, Unreal) and not build for engine sake.

I work in C# and have one assembly (DLL) for my "core engine", some separate assemblies for engine components and then one for each game. I have 3 active game projects using my "engine". I agree with others, having at least some basic game to work on is extremely helpful.

I almost always go breadth-first when developing these days. If nothing else, it makes it much easier to collaborate with others in my opinion/experience.