Agreed with what everyone else has said so far -- The first thing you want to concentrate on is writing clean, straight-forward, algorithmically-sound code in a high-level language like C++. In addition to being algorithmically-sound, you also want to be aware of how your data-structures occupy memory, and how your algorithms traverse that memory-space when doing work. Because processors themselves are so fast these days, they spend a great deal of time simply waiting for the memory system to catch up -- therefore, if you optimize your memory use and memory access patterns, you optimize your application.
After you have chosen a good algorithm and considered memory access patterns, *then* you may want to take further steps if the performance of your straight-forward code is not up to snuff. The first step here would be to look at things you can do sticking strictly to the core high-level language: making sure you are const-correct (which you should be using anyhow), applying 'restrict' to appropriate pointers, using canonical looping patterns, etc. so that you enable the optimizer to do the best job it can. If performance is still insufficient, profile and/or analyze the generated assembly and consider massaging the high-level language code where the optimizer has done a poor job (perhaps it isn't unrolling a loop that it could, or isn't pre-fetching the next operand). After that, consider re-writing the code using assembler intrinsic, and only then if performance isn't good enough would you want to consider assembly. In most cases, if intrinsics haven't gotten you nearly there already, going down to assembly isn't going to get you the rest of the way.
All of this effort should be guided by profiling -- you should never try to guess where your hot-spots are, much less what the precise root cause of them is.
Also, cast off any notion you might have about the "hardcore game programmers of yesteryear" that lived and breathed assembly, spending weeks and months to eek out 5% performance improvement. No one has time for that anymore, and even big studios with that kind of expertise available rarely go through the trouble.
That said, being able to read and think in assembly, combined with being aware of computer and CPU architecture, is a good skill to have. The Kip Ervine book is excellent, as are Randall Hyde's Write Great Code vol. 1, and vol. 2, and his other book, The Art of Assembly Language. In particular, the Write Great Code books focus not so much on assembly, but on getting the most out of your high-level code using many of the techniques I've mentioned above.