Uhm... you realize that if you'd simply use a map<long long, PeerData> you would have exactly the same result? Why are you so hell bent on getting a shared_ptr involved when it's absolutely pointless? This feels a lot like "new toy syndrome".
In fact, it would even simplify things, as your PeerData only uses the default constructor and a map will automatically create a default constructed object if necessary.
PeerData* data = &AllPeers[ipandport]; will do exactly the same, without abusing the map as an artificial crutch way to keep your shared_ptr alive.
The std::map was intended to be implemented regardless. I want to be able to group peers in different groups later. For example if 3 players are on Team1, the shared_ptr would be placed in a new std::map called Team1. Then I can let Team1 chat with eachother by just traversing that container.
I also want an std::map that states which connected peers are "Authenticated", such that non-Authenticated peers can be filtered.
All of these containers would hold the same shared_ptr, but they will be used for different purposes.
Question, map<long long, PeerData>, would the PeerData object be on the stack or the heap?