Frankly, I didn't give it much thought yet, but the first feeling towards your solution is kind of "rejective".
first, you say it yourself, it feels "crazy".
secondly : Bregma point number 1.
you are going to have to explain yourself about the "I didn't find it to be an enjoyable experience".
Also, please consider that some people just are on the edge of banning any usage of shared counted pointers (like google C++ guidelines) because it lacks a strong owner with clear memory responsibilities.
I don't advocate anything yet, I am using shared_ptr for help with RAII. and also avoid return by copy but still have return-result idiom and not "pass by reference an 'out' argument" which I dislike. (I don't have C++11 with move semantic so... sometimes I use RVO, sometimes shard_ptr for that purpose)
I find intrusive pointers kind of dirty, though I perfectly understand the motive (memory fragmentation notably).
my 2cts, though I didn't answer anything, I just thrown some random concerns around for you to take/reject/think about etc.