I've been looking at posts like this on how to manage resources like buffers and textures. It makes sense.
https://molecularmusings.wordpress.com/2013/05/17/adventures-in-data-oriented-design-part-3b-internal-references/

Currently I'm using shared pointers to Buffers and Textures. And my geometry objects hold those pointers. It means that if I delete the last model using a particular mesh that the reference count on the buffers will drop to zero and the buffer object will be deleted and the d3d resource has ->Release called. All automatic and safe. And no doubt very slow and inefficient.

If I switched using a handle system like in the article how do people go about managing textures for example?
I would have to somehow know that nothing is using a particual Texture any more and manually call mDevice->DeleteTexture(textureHandle).
So do people implement some form of reference counting inside the Device to count CreateTexure and DeleteTexture on each texture handle? It seems if I'm going to do that I might as well just use a shared_ptr anyway to count it?

EDIT: That might not be the correct link to the website, I wanted the one about using small integer handles. But I cant check because for some reason it's blocked from here...

Did you profile to make sure your method is actually slow and inefficient? If you make something that has an insignificant performance hit twice faster, you get an insignificant performance increase.

Well, std::shared_ptr generally uses a mutex when you copy or destuct them which never is going to be fast. But point taken...

I think it uses atomics over a mutex, at least in newer implementations of the stl.

You won't know what's slow until you profile your code.

That said, shared_ptr is threadsafe - it uses atomic variables for the reference count. That's an overhead you may not need.

The idea in the article you linked, as far as I can tell, is that you for example look up a Texture from a TextureCache using the handle and get back a raw pointer which is safe to use for that frame only. If you store or even worse delete the raw pointer then you will be in trouble eventually.

I would have to somehow know that nothing is using a particual Texture any more and manually call mDevice->DeleteTexture(textureHandle).
So do people implement some form of reference counting inside the Device to count CreateTexure and DeleteTexture on each texture handle? It seems if I'm going to do that I might as well just use a shared_ptr anyway to count it?

Many engines manage memory not in a totally dynamic way. Instead they define memory budgets for particular purposes and they define lifetimes a priory. For example, some assets are loaded at game start and unloaded when the game ends, i.e. they have game lifetime. Level based or region based lifetime are other popular methods. Streaming may also be used in which case buckets of memory are allocated once and then cleared and re-used in dependence on vicinity or whatever. However, the clue is that the lifetime is not measured but prescribed.

I do not know what exactly the Molecule Engine does, but the articles and comments in that blog mention such things, too.

After considering this some more I realized there isn't a code issue here, the issue is that I'm trying to design my game too much "bottom up" without considering how the low level compents will be used. it's clear now that I'll have some kind of "Level" class which represents one game level or world and that will load all of the ressources that items within it need when it is created and release them when it is destroyed. No real need to reference count or track usage. Just create everything on startup and release it on destruction. Other subsystems such as UI do similar with their own lifetimes.

I guess the issue here is the one that people mention. Trying to build an "engine" rather than a game without having the previoius experience to understant how the engine will be used. (Not building an engine on purpose, its just how it works out if you do bottom up design only).

shared_ptr is going to be slow if you pass that pointer by value everywhere, if you can pass by raw pointer (or in the worst case shared_ptr&) to avoid the atomic operations, then the only overhead left would be that your data isn't stored linearly in the memory, but for the most cases(for graphics resources) you don't care if it is.