store the server ip and the last time you heard from it in the vector instead, have servers notify the master server every 30 seconds or so and then simply drop or ignore servers from the vector if they have been silent for more than a minute.

Personally i wouldn't use a vector for this though, a map is probably better.

Is there a TCP connection between the listening server and the master server?

If so, you can remove the listening server when the connection is closed (return 0 or -1 from recv())

However, if the network connection is broken, or the machine is powered off or hard-crashes, the connection will "stay alive" for a long time before it's detected as closed.

You can fix this by sending periodic messages in both directions on the connection, or by turning on SO_KEEPALIVE (although SO_KEEPALIVE by default also uses a very long timeout.)

To map from an IP address (or socket file handle) to a connection object, use the appropriate data structure for doing value look-ups based on key. 99 times out of 100, that's a hash table. (In C++, an unordered_map)