That would completely kill C++ as being a pay-for-what-you-use / opt-in language. Most embedded systems that I've worked on still use raw-pointers, or a smart pointer that acts like a raw one, but only performs leak detection during development.
Owned pointers are a language semantic that have _zero_ overhead compared to manually using raw pointers and new/delete in the requisite places. Raw pointers would of course stay around for cases you don't want owned semantics. All an owned pointer does is call delete (or with std::unique_ptr any deleter policy function specified in the template) when the pointer goes out of scope. std::unique_ptr takes up the same space (assuming your deleter policy doesn't require extra space), has no additional runtime overhead (assuming optimizations are on and all the template goo gets inlined and removed), and imposes no additional memory allocation requirements beyond whatever your allocator uses. You can use unique_ptr with pooled objects trivially by allocating from the pool and using a release_to_pool deleter policy instead of the default policy. Reference-counted pointers, GC pointers, and other shared-ownership pointers are a different thing.
He referred to embedded systems where tools haven't matured and probably never will (because the specs keep changing all the time or nobody is actually bothering); and thus the assumption you mentioned "optimizations are on and all the template goo gets inlined and removed" cannot be made.
Furthermore, we're talking about video games here, and it gets hard to debug when your game runs at 1 fps because the compiler didn't fix the smart pointer's overhead.
Also smart pointers DO incur into overhead when used in more advanced cases (i.e. inside a struct hold by a vector inside a vector). This overhead may be possible to remove with C++11 (move semantics) but that's bleeding edge C++ that is still not as optimized as it can be; and not everybody has the possibility to use C++11 and must stick with older versions.