"The question also is: What will be more useful for business later? A small hobby engine that can never compete with the big ones, or experience in using the tools that everyone else uses too?"
Well, so long as the code is portable and meets basic criteria of general 3d engine, what is wrong with that? It doesn't need lots of features Unity has. Once you handle the basic lighting, a couple of cool effects, the rest is already game design. You need two things : a good physics engine and basic 3d renderer. It's shouldn't as hard as you think especially as an exercise before writing the actual content creation tools.
"But you also say you want to make tools for the game industry, so you do not want to make games in the first place, right?"
I want to do all of them. If I schedule my time properly I can create a sustainable business.
"In that case, a similar problem comes up: You spend years on engine and tools just to catch up with state of the art, and you might never get there, because Unity and UE already have all those tools and they improve them with a large team and also they are buying new stuff to add it as well."
My engine is already gonna evolve into a content creation tool later on even though it looks like an ordinary general engine. I am gonna use the first phrase as an exercise. Nevertheless, it is gonna become a good code base/library on which I can rely for my content creation tools. Even if it is not gonna be as good as Unity, the journey is still gonna be instructive and it will be a good preparation for writing actual tool that would make the difference.
"In the future such 'stuff' may be AI driven character animation library, or AI driven procedural content creation tools. Who knows?
So if you want to sell tools for game industry, you likely have to work on something like this. Something that does not yet exist. Nobody needs another basic physics engine or another state of the art renderer. You'd need to address open problems. Pick one, work on it for years, and with luck you get something working before all the others and they might be interested. Chances are surely tiny. I do not want to sound demotivating - i do exactly this, at big risk."
I will incorporate AI based content creation tools into my project as I've already mentioned in another topic days ago. The actual thing everything might revolve around.
"That said, i have the impression you want to do a bit of everything but you have no clear idea yet on what to focus on. Games or tools? You can do it all, investing much time, learning a lot... i propose you continue with this, but to make a living out of it, at some point you need to break out of this, have an idea, a good one. So start thinking about it already. ... and be ready for plan B which has nothing to do with games as well."
All of them. Both games and tools. Plan B may be web development but I will try to do my best to stick with game development. Thanks for the recommendations.