I'm working on a game engine in C++. My current approach is described below:

• Engine is a library, provides all the necessary shared functionality (rendering, physics, etc.)
• Games are executables that link to the engine
• Editors are also executables, also linking to the engine

My initial plan was to work with a very specific constraint to avoid feature creep: games just need one window where everything happens, so we just need the engine to create this window (e.g via SDL), handle the main event loop (user input, etc.), update its own subsystems, and then update the game logic (which then queues up new tasks for the engine, e.g rendering). If the engine gets told to stop (main window closes, exit command on console, etc.), it shuts down the game as well.

Note: you can sort of picture it the way Unity works with MonoBehaviour objects, except in this case the entire game is one big “MonoBehaviour” that the engine queries for tasks.

The above approach works well enough to handle just the engine and a game hooked to it, as the latter can now focus entirely on game-specific code. On the other hand, this does create problems when trying to connect the engine to an editor, which tends to have a much more complex GUI. It would be unnecessarily complicated to write some kind of “full GUI abstraction” in the engine, just to allow the editor to create all of its UI through it. Thus it would make more sense to have the editor application handle window management, events, etc. and use the engine as a “subsystem”.

• For example, in an editor, we may now have several windows we want to render to, and at the same time, we don't expect to handle inputs from said windows the same way as we would in-game
• Same applies to the update mechanism: we don't necessarily want to update the entire engine on every frame while editing, and some parts of the engine would be turned off altogether

With this in mind, now it seems like the better approach is to have the applications that use the engine also handle the GUI and the main loop, which dictates what the engine does every frame. Few games need special GUI management, however, so this would cause a lot of code duplication, and it can be advantageous to give the engine tighter control over both the GUI and the update logic. Overall, it seems like a tradeoff.

Is there a way to have “the best of both worlds"? Are there good case studies you would recommend?

@Thaumaturge You're absolutely right, and I apologize, I worded my post in a misleading way: by “UI” I refer specifically to OS window management (I am not sure what the exact terminology is), i.e creating actual windows, dialogs, etc. using the APIs provided by the operating system (older apps would use something like Win32 API on Windows, nowadays you can hide the ugliness via libraries like SDL, Qt, wxWidgets, etc.). This is something that's highly impractical to try and expose as an API on the engine, it only worked fine as long as I had the constraint that “the game only needs one window where everything happens”.

For what I would call "in-game GUI", i.e widgets and other 2D elements within the game's main window, I would indeed just use a 3rd party library (in my case, I planned to use ImGui), possibly combined with some homebrew code if a user wants to tweak the aesthetics a bit more, e.g for menus (raw ImGui favors function over form, so it doesn't look very flashy).

Now technically it would be feasible to use the “in-game GUI” approach to develop an editor as well, but this would limit the capabilities of tools to what the engine can provide. By contrast, outsourcing the responsibility to the editor application would remove a lot of potential bloat from the engine.

As I thought about it more, I think my best bet is to avoid overengineering things and apply YAGNI. So I can extract the whole “runtime” (GUI, I/O, console, etc.) since I'm likely to reuse it between games, and if a game wants to use it, they link to both it and the engine. If a particular game implements “nonstandard” frame management (w.r.t the one provided by this runtime API) then it can just interface with the engine library directly. Same goes for editors, which are likely to use their own GUI management anyway, and therefore should not be bound to something the engine tries to default to.

yah-nosh said:
Is there a way to have “the best of both worlds

For me, the main reason i would not use off the shelf engines is that they are not modular. If i want to replace the physics engine with something that actually works, i'm already better off with starting from scratch. For people with such mindset, there is not so much around.

So personally i would offer a GUI system, which you'll use for your editor, but can be included to the game as well, if people want so.
I would be also fine if you would use just ImGUI for your editor for example, and for a game i have to make my own GUI to look nice. I don't expect an engine to have absolutely everything.

Raylib seems an example of modular / optional systems. Never used it myself but looks nice.

@JoeJ I aim to limit the responsibilities of the engine so it doesn't take forever to implement, but at the same time, it's always nice to provide some extra utility for games, since otherwise there's not much point to making the engine in the first place. That, and it's also a good idea to define a scope, since no engine can be “infinitely modular”.

In my case, I intend to support a subset of 2D and 3D games (adventure, FPS, etc.) since these are the games I intend to make. I would not support something like an RTS though, which would probably need a different design altogether.

I do make sure to keep all my modules behind an interface so they could be swapped out later (e.g replace the physics library with a different one), but always with a specific scope, so I create an API that “does what the engine is designed to support, and nothing more”.

yah-nosh said:
no engine can be “infinitely modular”.

Sure. But i guess this goes both ways. If i would know U engines well enough, i could probably replace it's physics.
Tbh, i probably just don't want state of the art tech in general. I want to invent new tech, and then see what it gives regarding new games.

yah-nosh said:
In my case, I intend to support a subset of 2D and 3D games (adventure, FPS, etc.) since these are the games I intend to make. I would not support something like an RTS though, which would probably need a different design altogether.

Sounds you want to use the engine mainly for yourself, so my feedback is pointless.
Then you can do what you like. Which is the only proper way in a creative industry anyway, imho.

But then i would not worry about the GUI topic until you need it for a actual game. Maybe the game requires very little GUI, and making such specific GUI is little work then. And eventually it evolves into a general GUI system by reusing it for later games.
Contrary, if you try to make a general solution now, that's lots of work. And when it comes to the game you may still need to add specific stuff or do unexpected changes.

Sounds you want to use the engine mainly for yourself, so my feedback is pointless.
Then you can do what you like. Which is the only proper way in a creative industry anyway, imho.

I have a specific goal in mind, but I'm still open to other people's input. Also, while I am currently developing it on my own, I do intend to make the engine “dev friendly” enough so I could invite others to collaborate at some point.

yah-nosh said:
I have a specific goal in mind, but I'm still open to other people's input. Also, while I am currently developing it on my own, I do intend to make the engine “dev friendly” enough so I could invite others to collaborate at some point.

Precisely. Applies to me as well.

But i'm not really sure if this can actually work in some standard best way. And we are a creative industry, with our duty seemingly being to replace rock stars. Or at least their results.
And now this makes me think: A key of success for rock stars was to give a fuck about standards. Maybe we can learn from that, and ‘do what you want’ is indeed good advice.

Well - not sure of course. And sorry for diverging from the actual topic. ; )

yah-nosh said:

My initial plan was to work with a very specific constraint to avoid feature creep: games just need one window where everything happens, so we just need the engine to create this window (e.g via SDL),

[…]

On the other hand, this does create problems when trying to connect the engine to an editor, which tends to have a much more complex GUI.

The funny thing is that you mention SDL, and they struggled with this problem for years, before fixing it in SDL 2. I think for a while they expected you to set the SDL_WINDOWID environment variable and they'd render into there instead of its own window.

I would suggest that, rather than taking full ownership of window creation, you'll want to make your library able to either render into an existing window or to take some sort of callback for window creation, where you can provide a default.

As for the event loop, I think you're potentially in for a lot of pain there if you wrap that entirely and don't forward events to the places that need them. I'd advise working right now on trying to integrate your game into a minimal Windows application and making sure it does what you expect. For example, just have a rendering window and a button both created by the application, and verify that you can render into that window AND that the button is receiving events and can act on them.