For me, the key is to spend time on it in a systematic way, so it stays part of my life. It's amazing how much progress you make even if you spend only a little time on it daily.
Getting to the point where you left off last time is the difficult part here. For some aspects it takes time in itself (which is not bad, as it's tracing your steps, checking them again for correctness of decisions), for other aspects, you can start in seconds with just one look.
Pretty much this... if you only have 4 hours per week to spend on game dev, so be it. Just make sure you try to spend your 4 hours EVERY week... or at least as often as you can.
Those hours add up fast, and expecially when learning something new, skipping a week / scheduled time is the worst thing you can do. For one, every week you are not doing your scheduled "training", it becomes harder to get back into training... just as with PT for us civilians. And also, you will forget a lot if you do not keep using it.
The hard part is to start, when you lack confidence and direction as well as skill and expierience, and then from time to time when you have to solve a difficult problem that cannot be done in one or two weeks. Push through these though times, plan smart (maybe make sure you stretch the hard stuff to multiple settings so you have spent at least half of your weekly game dev time on something fun), and you will have a lot of fun with game development.