but I feel that if I become great in Unity, I will only be able to create a game in Unity and in nothing else.

Not only is this true, you will not understand what is going on in all those black-box components, making it difficult to fix problems.

My advice:

There are different ways you can go, depending on what you want to learn. One area of study is 3D graphics. How do graphics cards work? What is really going on? To learn about all this stuff you can make software renderers from scratch. This is not easy, but you will learn a lot.

Libraries like SDL, SFML, freeglut, GLFW, and DirectX give you a framework for low level access while handling all the common things all games need. DirectX is a little different because it only works on Windows and doesn't give you a window framework, but making a generic game window from scratch is easy and well documented. The other libraries are all for OpenGL, and work on any platform, handling the window code.

All of these libraries give you low-level access to rendering, input, and sound, which is all you need to make a game. If you are interested in learning how these libraries work, the opengl ones are all open source. You could write your own, or review the code to see how they do things. You will learn a lot from reading their code, so don't discount that.

The third option is to use one of the above libraries and make some games. None of those libraries are hiding anything like the game engines. But after using them at a low level you will begin to understand why those game engines exist. There is a lot of framework code to write, debug, and optimize, and every times something changes you have to go back and check everything again. For one person making a game this can be too much. But there is nothing wrong with doing it yourself.

Hopefully that gives you some ideas so you can pick a direction, research it, and ask some more specific questions.

Actually, I'd start with this one:

The Black Art of Multiplatform Game Programming

by Jazon Yamamoto

Link: http://amzn.com/1305110382

The "Game Engine Architecture" book is good if you want to make a game engine like Unity or Unreal. I don't know anything about the game patterns book, but with so many reviews it looks great.

That book I recommended is new. It came out the same time as my book, and it has done really well. I think the title is bad, because is doesn't stress the fact that it teaches game programming with SDL.

You question really comes down to what you want to do... You either want to make game engines or you want to make games. If you want to make games, you should be using an engine instead of trying to recreate the wheel and most certainly will be spending 98% of your time making the engine and probably never see a game since engines can take years of a single persons life. If that is the route you want to go that is awesome, but I guess if you don't know you should start delving into the abyss of understanding lower-level graphics programming.

If you want to start off small to wrap your head around more of game programming from a mid-level I second the suggestion for SFML I have used in many many times, I even started my own DirectX version years ago, turns out that is when I realized I wanted to be a game programmer not a game engine developer. So maybe this is something you have to do to figure that out as well. Good luck to you hope my advice helps you out.

You might be interested in Handmade Hero.

If you write everything from scratch, then technically you've made a game engine. But if you don't try to make that code work for other (abstract, non-existing) games but instead just try to get your game to work, then that doesn't really matter. The bad thing about trying to write a game engine is that it takes away a lot of time and effort from finishing the game. In the end, you'll most likely not finish the game you were trying to create in the first place.

I wouldn't recommend writing absolutely everything from scratch, if that's your intention. Think about what you want to learn to do and find ready-made libraries, sdks etc for the rest. There's no shame in relying on the work of others for parts of your game. And some of these things are real time wasters, you can spend weeks getting some tiny feature to work just right on a specific computer while others have already solved that problem and provided the solution for you to use.