Technically, to build a game and engine from scratch, you have to start by inventing the universe.

Unity is great- it gives you a top down approach rather than bottom up. It lets you make a game at the start, and work your way towards the arcane knowledge. You may never have to know how to program your own graphics pipeline, but understanding it will inform what you do and make you a better developer in the long run. In the meantime, you can work on the OTHER side of game development- the arcane process of building high-level gameplay from building blocks. I like to think of it as super-advanced lego, that unlike lego bricks, could actually feasibly used to build real "house" in the future.