The peer->data member is never sent anywhere. It is just a pointer in the peer object where the application can put data.
Why are you storing things in a packet and not sending it?
Taking a closer look, the PeerData is deleted when the case scope ends, because the shared pointer that owns it falls out of scope and there no other shared pointers referencing that data. Attempting to read from peer->data later segfaults, since the data has already been freed.