You really should use some kind of library(I'm no python expert by people say Pygame is good). Trying to write games "from scratch" can cause performance issues if you don't do it right. Most modern engines/SDKs are written for optimum performance and use hardware acceleration to boost your graphics performance. Using an engine does not cap your ability to have full control, and if you do see something you want to change, you can just modify the engine to your needs(as long as it is open source).

You should us a media library of some-sort, so that you don't have to write os-specific code to get a screen, along with input, and the ability to play basic sounds.

I actually have a simple media library that I developed for the latest version of Python: https://github.com/GoranM/pslab.

You can actually see me use it in the later parts of my Python video tutorial series:

If you're looking for something with more features, you could give the Blender Game Engine a try. I actually did a bunch of video tutorials for that, which you can find on my youtube channel.

Good Luck!