While client A and client B will have the same IP address, they will have different source ports.
Think about it: If there was no way to tell them apart, how could the router know which return packet to send to which client?
Thus, you can simply identify the clients with their remote IP:port tuple, just like with any other clients.
Lol, that was really obvious. I feel stupid now lol.
Thanks a lot hplus.
EDIT:
2nd problem solved.
In case anyone find this with the same problem I had, you need to start the addrlen with sizeof(struct <your address type);