I would say that in a nutshell, game programming consists of doing simulations. If you can program a simulation of a ball bouncing, you can also program games.
A ball-bouncing simulation in it's most basic form consists first of all of a description of the current state of the system: The location and velocity of the ball and the position of any walls. Then at regular intervals, the state of the system is updated by moving the location of the ball a tiny bit according to its velocity and time elapsed since last update, taking any necessary collisions with walls into account by reversing the velocity of the ball along the normal of the wall. After that the walls and the ball at it's current location are drawn to the screen.
The repeated application of updating the state of the simulation and drawing will make the apperance of a ball bouncing around on the screen.
Now add in an additional step of sampling an input device and using that information for moving the walls in the simulation and you've essentially created a very basic pong game.
Any other game will at some abstract level also consist of those same concepts
A description of the state of the game
Sampling of input
Updating the state according to input and rules of the game
Rendering the state to screen and other output devices.
The only thing that differ pong from the latest multi-million dollar title is the scale and complexity of how those things are done and the specifics of what is stored in the state and what rules are being applied when updating.