Of course, using shared_ptr for automatic resource management can be very convenient. However, he asked for efficiency, and shared_ptr is not the most efficient way to maintain pointers, especially because shared_ptr is thread safe even if you don't need it: The reference counters are always synchronized.
This is certainly correct, and something to always keep in mind when copying around shared pointers. Passing around a shared or weak pointer is not close to a no-op as compared to passing around a raw pointer.
However, in the context of an asset manager you need to consider that the rather slow atomic increments/decrements used in the shared pointer are entirely neglegible compared to only a single disk access for loading an asset. We're talking of 30-50 nanoseconds versus 8-10 milliseconds.
Formally, giving out a weak pointer as suggested by Mnemotic is the correct thing to do (though giving out shared pointers would probably work just fine 90% of the time too), seeing how the manager owns the resources and should be allowed to toss them on a as-needed base, but safely so no resources "disappear" while in use.