I ended up writing my own smart pointer for resources.
I'm only mildly happy with it. There might be errors in my reference counting shenanigans but I haven't noticed a memory issue yet, though I haven't put it through the wringer yet. And there's probably some C++ sloppyness in there but hopefully the idea gets across.
It functions like a shared_ptr<> because it reference counts. But it lets you handle the storage of the asset. So if you wanted to unload assets until they're used again you could do that on the back end without uses of Asset<> ever knowing. Since its a template, it duck types a LoadFromFile () static function on the type you pass to handle loading. Eventually I want to add in things like loading on another thread and using a dummy asset until its complete. And automatic reloading of modified files. It can all be done behind the scenes.
I have classes like "Texture2D", "Mesh", "Shader" and these all provide a LoadFromFile (). I simply call Asset<Texture2D>::Load("Somefile.png");
I have a special function if I have to use a generated asset.(Such as a procedurally generated texture.) I can either give it a name and enter it into the registry of assets, or I can "wrap" it which creates an Asset<> who's job is not to delete the resource its referencing. Instead its some external object's job to delete it. This is the part I'm not entirely comfortable with, but it unifies the two methods of resource containment. (File Loaded and Generated).
I can't pass extra arguments into LoadFromFile (Though this could be solved with a variardic template). There is global state here, but I suppose it wouldn't be hard to have some sort of "AssetCache<T>" instead of using global state. Also I'm using a std::map<> because I was lazy which can be slightly slow depending on how picky you are, however I don't call Asset<T>::Load() every frame. However if I have some sort of rapidly created and destroyed object that initialized itself by acquiring an asset I could hammer my std::map. Like I said I'm only mildly happy with it but I thought I'd post it in case it provided some sort of idea, also if Its a horrible idea I would love to know too.