From experience, it's surprisely a combination of both!
For example: you will learn the skill to set up the game canvas and adding your game components to the game canvas to put together your game.
Even the simple game like an arcade shooter clone in Java will make you recall all the basic intuitive math! Even classical physics kinematics equations allow you to make your game super smooth! Everything you learned from your introduction to Java programming are all important. Do not neglect learning any of the concepts. They all at some point will play a role. (ArrayLists are super useful so you don't need to keep track of the size of the arraylist yourself!) (Rectangle objects are super useful for basic collision detection between your game objects)
A programming language like Java is huge in the functionality it has to offer. Even I am constantly learning new classes in the Java API to make me more productive in my game project.
Honestly, when I learned game programming. It took me a month(on and off) to make sure all the basic meat of the game was there(bgm, sound effects, collision detection, life bar) and also everything was bug-free. A month of time getting everything to be bug-free was one of the happiest thing I did in my life.
Also, please get started! This is the only way to test your skills of what you know and what you don't know. Reading books alone will only get you so far. So you actually have to do a bit of reading and start applying the knowledge immediately of what you read.
I agree, start small. I heard people constantly want to make the next-big thing but end up giving up because it did not meet their expectations. Think of the game project as a hobby that makes you want to learn, make you curious and make you think as a human being. Just start off putting a game window on the screen and then start adding game objects. If you don't know or understand something ask, there are talented and smart people on this forum willing to help. Just be sure to post what you have done so we can better assist your problem.
Please bear in mind, don't be put off by the complexity of game programming. There will be things you will need to research to help guide you to your solution. Struggle with the coding and you will find programming to be very elegant.