This has become a rant about shared_ptr, not smart ptrs in general?
We use intrusive ref counting to eliminate the extra heap allocations. Our pointer wrapper can be defined to be either thread safe or not (by default it is not).
I agree that smart pointers can cause performance issues, mostly with cache misses since you have to fetch the object (or ref counter location) just to copy the pointer address.
It would indeed be wise to use raw pointers for low level systems such as rendering. The render manager may store a single smart pointer as objects are registered. This will preserve the object while it is inside the renderer. Then, internally raw pointers can be "safely" used.
Smart pointers also often cause circular references and memory leaks. But imo, their benefits FAR outweigh their pitfalls.