Well, playing through Chrono Trigger 18 full times in one summer kind of cemented that I wanted to make video games for a living.
After that I was always interested in programming taking high school programming classes (Pascal and something called RealBasic, which was a Mac WYSIWYG sort of editor that used Basic as a scripting language. It was pretty cool and essentially handled the gameloop (or update loop) for you). The engineering classes I took in high school had us interfacing Basic code with circuits we built. My final project was a wooden arm that you wore and moving it moved a 2D articulated arm on the screen. I never really spent too much time out of class working on programming even though I always intended to.
After high school I took a CS degree, met a bunch of friends, learned to program together, built a few crappy games together (on our own time), we all graduated and now have worked together at the same studio for the last 8 years or so (through referals we pretty much got each other hired). Lesson: make friends and keep in touch because referals are the best way to get past HR and have an interview.
As for the question of being frustrated not having the time to learn everything I want? Of course! But a bigger problem is I don't have the time to get the things I already know how to do done! Not enough hours in the day now that I have a kid.