I have a TCP server listening, and hundreds of clients connecting to the server.

Whenever a client is connected, it sends a unique ID (say "123") to the server "X", the server "X" authenticates the unique ID "123" against the database, and if it checks out, it stores another entry in the cache to indicate that "123" is connected to server "X". This is so I can put many instances of the servers and know which client is connected to which server.

When client disconnects, the server removes the entry from the cache.

Now I am running into an issue that if a client is connecting/disconnecting all the time, there is a race condition when the event is sent to the cache. I am observing that if a client connects and disconnects repeatedly, it may show as disconnected on the server side. The disconnect may take several minutes because of timeout, flushing out and closing the TCP stream, etc, that the new connection is posted first, only to be removed later by the old connection.

How do I handle this flow to correctly store client's connected status?

Send ACK packets to mark if a client is connected or disconnected.

The heartbeat from the client (or from the server on behalf of the client) is only half the equation. The other half is a dedicated process running on the database server that can occasionally connect to the database and timeout clients, and potentially perform other maintenance tasks. The exact implementation depends on the application, but at the very least you need those two elements.

Well, when I said server, I actually simplified what we really have.

This is closer to what we have.

CLIENT <---> TCP SERVER <---> RECORDS SERVER <---> DB/CACHE

Client talks to TCP Server, which is only to manage active TCP connections. TCP Server isn't aware of any DB/cache. TCP Server talks to Records Server for authentication and posting the 'connected' status.

TCP tells Records Server that "hey, user X is online, and it's connected to me"

So you can increase the number of TCP servers independently of the Records Server.

Everything else in the cloud will talk to the Records Server to determine if user is online.

I changed the way we store this connected status to include the client's IP address as part of the check. If the disconnection isn't detected until later, it won't remove the current connection since the client's port will be different (at least for our case, the client isn't set to one port). It alleviates a lot of the problems now, but I'd like to hear better options.

I know of a company that did that, and scaled it up to a surprisingly high number of users. That being said, it is tremendously wasteful of machine resources, AND it is slow to respond to client disconnect. Using specific solutions for presence is a better user experience and a cheaper, more reliable way to run the service. For medium user counts, a pre-existing tool like ejabberd can do this for you. For massive user counts, you're going to end up writing/re-writing this on your own, because everyone ends up doing that (Google, Facebook, Riot, Steam, as well as us at IMVU.)

Assuming all your services can be connection-driven, like presence. But that's often not the case. We have a handful of RESTful services in our game, for instance, that require the database to be "pumped" every few seconds or so. It all just depends on the full extent of what you're trying to accomplish