My personal biggest headache about modern engines is that they are just insanely gigantic and maze-like to work with. Though I suppose that would be true of any sizeable engine, but you can tell that many hundreds if not thousands of people have layered workaround on top of workaround into a lot of the code. My main experience with UE4 has been spending dozens of hours tracing through code, googling or reading documentation to try and figure out where something I want to change is even defined in the engine code and what methods I have to get at it or override it. Even dealing with blueprint there is a lot of situations you run into where it's something to the tune of "Oh there isn't a node for that, but there is a C++ function buried somewhere that will let you do that! Connect them!"

Convenience seems to be a double-edged sword in many cases, it's quick and easy to do many otherwise hard or time consuming things in the engine due to the robust tools and deep systems set up already for you, but it makes learning it and changing things that don't just work for you right away at the top level a heck of a job.

It's hard to say whether you would have been more productive, personally, I think making an engine yourself of decent design and depth sounds like a great learning experience if nothing else.

3 hours ago, Satharis said:

My personal biggest headache about modern engines is that they are just insanely gigantic and maze-like to work with. Though I suppose that would be true of any sizeable engine, but you can tell that many hundreds if not thousands of people have layered workaround on top of workaround into a lot of the code. My main experience with UE4 has been spending dozens of hours tracing through code, googling or reading documentation to try and figure out where something I want to change is even defined in the engine code and what methods I have to get at it or override it. Even dealing with blueprint there is a lot of situations you run into where it's something to the tune of "Oh there isn't a node for that, but there is a C++ function buried somewhere that will let you do that! Connect them!"

Yeish, I get a head ache just thinking about it.  I opened up unity after downloading it for Windows, then I quickly closed it and have yet to open it again.  I dread the day I have to open it and begin the long journey of learning how to manipulate it.

So, I've been on a similar journey. Off and on I've been building an engine for about 5 years. Originally in DX11, then ported to DX12, and most recently I've rewrote it from scratch in Vulkan, though I'm just at an early stage now. I think I've learned a lot, and it was a worthwhile experience, but I'm nowhere close to finishing a game after all that time. In that sense, if I wanted to finish a game, I could have done so with Unity or Unreal in much less time. However, I'm sticking with it because I have an original idea I'm not sure can be done right now in the off-the-shelf engines, and I want to give it a shot.

That said, I have evaluated the commercial engines, mostly Unity and some Unreal, as well as quick tests in less popular engines like CryEngine, (the now defunct) Stingray, etc. I've found they all did some stuff well and some stuff not so much. Unity is pretty solid overall, and probably would be my pick if I had to choose one. There are some specific reasons it's not perfect for my project, and I would like a stable Linux editor, but overall it's OK. Unreal looks really nice graphics-wise, but I found the editor to be bloated (it takes forever to load, even on a top-shelf system), and is really unstable. I've had it crash when doing seemingly common things like deleting an asset from the project (and made worse since it takes forever to load up again). One time the crash completely corrupted the project and I was unable to open it again (loading the project itself would crash). Of course, I had a backup, but it still doesn't make me feel comfortable that my project could be permanently corrupted just by removing a texture. Unreal also has limited support for custom shaders, which is fine if you like the "Unreal look", but if you want to do something unique there may be hurdles. The smaller engines I tried have some interesting features but ultimately didn't look significant better than Unity/Unreal. And there is a concern of continued support with the less popular options, like the financial situation with Crytek, or Autodesk shutting down Stingray after like 2 years. That would be really troublesome, for example if you spent 2 years building a game in Stingray only to have the engine maker abandon the project.

However, if your game is fairly generic in the functionality and graphics style, you could certainly do something with Unity or Unreal. In terms of the level editor, you wouldn't be using the engine editor to give to users, rather you would build a custom editor into your game itself. In which case, you can make it simple and user-friendly. Both Unity and Unreal use PhysX, which is pretty decent for most common use-cases. You can use arbitrary convex shapes, though, to your point, Unity used to support concave objects with an older PhysX version, and then Nvidia removed that feature. While you can still simulate concave objects with a set of convex shapes, it would require some rework if you had built your game around this functionality.

In my case, I'm investigating advanced physics solutions using the GPU, and this is a little difficult with commercial engines. While it can be done, it feels like you are fighting the engine in a way when attempting this. For example, in Unity, you could write a physics engine in a compute shader and then render dynamic objects. However, if you do this, it bypasses all the standard material shaders, in which case you lose shadowing, lighting, etc. So you'd have to rewrite the lighting and shadowing yourself and then hope it plays nice with other objects that are rendered normally. Unreal, on the surface, seems better since you have the source code and can make arbitrary changes. However, if you want to update the engine, then you'll have to manually merge those changes every time you want to update, and things could break in unexpected ways. At that point, you would have to have extensive knowledge of the code-base and would probably be capable of writing your own engine.

All that said, if you are making a vanilla game, then the existing commercial engines would probably do fine. They have their pros and cons, but they cut out a ton of work for you on day one. If you are trying to do something more innovative, I think custom engines still have a place. Even if not, working on one is a great learning experience and definitely a solid way to gain some skills. I've gone back and forth over the years (and abandoned and restarted the project several times) but I do think it can be worthwhile in some situations. Good luck!

9 hours ago, Satharis said:

Convenience seems to be a double-edged sword in many cases, it's quick and easy to do many otherwise hard or time consuming things in the engine due to the robust tools and deep systems set up already for you, but it makes learning it and changing things that don't just work for you right away at the top level a heck of a job.

I would definitely have this concern with a commercial engine.  I use a lot of Java SWING in my job and even getting that to do what I want can be a pain at times.  For my game editing tools, I developed my own GUI component toolkit in pure OpenGL, and while its certainly a lot more limited than what SWING offers, at least I can always easily get it to do what I want.

3 hours ago, cyberpnk said:

Both Unity and Unreal use PhysX, which is pretty decent for most common use-cases. You can use arbitrary convex shapes, though, to your point, Unity used to support concave objects with an older PhysX version, and then Nvidia removed that feature. While you can still simulate concave objects with a set of convex shapes, it would require some rework if you had built your game around this functionality.

The biggest challenge isn't the shapes or detecting the collision, but handling how the game reacts to the collisions.

For example, when the player collides directly with a wall, he normally bounces back from the wall (which is what you expect).

However, if the player "clips" a wall at an convex corner, then I do not want the player to bounce back.  Instead, I want to adjust the player position so that he isn't clipping through the parallel wall of his movement direction, and keep his momentum and movement direction.  The idea is that my game favours the ability of the player to "slide" along walls from a corner rather than bounce away from a corner.

Yes, it isn't realistic, and it might sound a minor detail, but it makes a huge and important difference to the feel of playing the game.

3 hours ago, Martin Brentnall said:

However, if the player "clips" a wall at an convex corner, then I do not want the player to bounce back.  Instead, I want to adjust the player position so that he isn't clipping through the parallel wall of his movement direction, and keep his momentum and movement direction.  The idea is that my game favours the ability of the player to "slide" along walls from a corner rather than bounce away from a corner.

Yes, it isn't realistic, and it might sound a minor detail, but it makes a huge and important difference to the feel of playing the game.

You are correct fully bouncing off walls is not the proper thing todo. I have implemented “Collision Detection with Sliding”, but it was a while ago and it’s way past my bedtime so sorry I can’t remember any details, except it was pretty easy todo. IIRC I just had to tweak a couple of functions ... the answer might be nearer than you think.

As for your original question - no I don’t think you made a mistaking implementing your own engine. 

When I tried to understand Quake, Doom, Unreal, Unity ... I struggled to get anywhere but when I got neck deep in my own engine I gained the deeper understanding and insights I previously lacked. Now I can understand those other engines and finally could use them if I really wanted to. I suspect you’d be in the same boat so all is not lost.

So yeah go for it I reckon doing your own engine is great ...