It is the difference between acting wisely versus blindly following a mantra.
The mantra is generally correct. It serves as a simple reminder of a larger concept.
There is a big difference between making obvious, direct, simple improvements such as removing duplication, versus what the original talk was talking about 'worrying about the speed of non-critical parts of the program'.
The first one is appropriate, often given terms like 'refactoring', 'paying technical debt', or 'code clean up'. You see something that is messy, a duplicate calculation, some loop-invariant code, or some easy tree pruning, and you fix it when it is noticed. That is not premature, that is the perfect time to do it. It doesn't matter if the compiler might have caught it, the change is an improvement. No worry or stress is involved. This is a great option.
The second one is common to see. In fact, we had one in the For Beginners forum earlier this week. The quote is about worrying and fretting over something that may not even be a problem: "I think my program might potentially be slow, I don't know if it will be slow, and if it is slow, I don't know where, so I need to write crazy things that I don't even know if they are faster at every step." Might as well start hiding in a cave now because there might be a zombie outbreak someday. That falls under the 'premature' umbrella. That is typically a bad option.
The quote also talks about the small number of things that you expect from the outset might need some performance tuning. Your first pass at an A* search algorithm that will be used by thousands of game actors? Yeah, anyone can tell you up front that is going to need some care so it doesn't explode either in terms of memory or computation. A particle system where you want up to a million active particles running at 120 FPS? Yeah, that's going to take some careful thought. But even in these cases you make a plan, implement the plan, and then test, measure, and compare like a computer scientist.
