Today I gathered my group together in which we had 6 of us trying out the networked game play. It didn't go too well. When there is only 2 of us everything works as expected and it's enjoyable. However, some strange discrepancies started occurring when there were 6 of us; 3 or more really. Individuals started to time-out mainly, some were even crashing for some unknown reasons. Mainly tho, the time-outs is what I'm concerned about. That isn't supposed to happen unless they don't receive a reply from the server after 30 seconds or so, or if they disconnect manually.

One of the strange things I noticed during the test on the Server log was multiple requests for the particular individual to exit the game, which was very strange because they weren't requesting to do so.

My network system is currently just TCP, which I think may be the root of my problem. I've been thinking that all of my packets should be guaranteed to be delivered timely since we are only working across LAN, I'm starting to become doubtful. However, another concern I suppose is maybe I am sending update messages to frequently.

Right now every .125 seconds and if the player's transform changed I send a message that looks like this:

// send our transform information to the other players
  transMsg << NetworkManager::GetInstance().GetServerCmdFromMessageType(MESSAGE_TYPE::Game)
           << "!SPECIFICOBJECTMESSAGE " << netCompRef.GenerateMessageID("UPDATEOBJTRANSFORM")
           << gameObj->m_Pos.GetX()   << NLDS << gameObj->m_Pos.GetY()   << NLDS << gameObj->m_Pos.GetZ()   << NLDS
           << playerCam->m_Rot.GetX() << NLDS << playerCam->m_Rot.GetY() << NLDS << playerCam->m_Rot.GetZ() << NLDS
           << gameObj->m_Scale << NLDS << "\r\n";

I'm also doing something not good for when we are shooting. Every bullet that is generate will send to the server the creation of the bullet, which since we have rapid fire now is definitely sending a ton of messages to the server, then back to the other connected clients. Also, another thing I was thinking is maybe my router's firewall is detecting the influx of packets and is thinking it's a DDOS or something, which then terminates the connection on that end. However, I don't think that's the case, but it might be possible.

Thanks for any insight in helping me improve my networking, I'm willing to hear any suggestions.

I've been able to reproduce the random disconnect bug we were encountering earlier today. It seems whenever more than 2 people join, 3 people in my test case, one of the other clients will get dropped on the server end, but only after an arbitrary amount of time has passed. I was monitoring the time since last ping on my server end and that's never reaching the time-out period. Something else in my server code is doing something strange somewhere. I really like the idea of reducing the size of my sent packet, I am in the process of working on a way to register the particular network messages as enumerations, also to reduce my string comparisons. I hope that will help also.

Rule #1: If you can't measure it, you can't control it. Your first order of business is going to be to get a precise accounting of three things, in this order of importance:

1. What packets are being sent from the server?
2. What packets are being received at the clients?
3. What's happening to the packets in between?

The first two you track the same way: Put code in your networking functions to log all your activity. You definitely want to log the timestamp (frame number is probably good enough) and number of bytes of every individual receive and send; secondly you want to log the message types (so that you can see what kind of traffic is causing problems); thirdly you want to log object IDs (so that you know what specific objects are causing problems); and finally if your logging system has the capacity, it's certainly nice to see the full message contents in the log. But above all you need to make sure that the logging itself doesn't slow your game down and contaminate your data--you need to be smart about logging to memory and committing that to disk in block writes infrequently and asynchronously.

The third you track by using tcpdump or Wireshark to monitor the actual packet traffic. But that's third on the list for a reason--that's probably not your problem. Focus on the first two for now, and just know that you've got options for the third if those don't turn up anything interesting.

I won't randomly suggest things to try in the absence of data, because Rule #2: Premature optimization is the root of all evil. However, here are some things that your logs might reveal that would be problems... but don't rush off and act on any of these unless/until you have data that shows they're your problems:

• Maybe you're only calling receive once per frame. When all your data generated for a given frame fits within a single MTU window (probably 1400ish bytes), everything would work fine. The moment it spilled over into two+ packets per frame, you'd simply fall further and further behind with every single frame.
• "One of the strange things I noticed during the test on the Server log was multiple requests for the particular individual to exit the game, which was very strange because they weren't requesting to do so." As you've described it--commands are being sent and you don't know why--this sounds like a bug, and is certainly a red flag. In principle, you should know exactly how every packet in your log got generated.
• "My network system is currently just TCP, which I think may be the root of my problem." This is exactly the kind of assumption you should not leap to in the absence of data. Your initial thought--that a modern LAN or wifi connection should be able to transmit data fast enough and reliably enough that there's basically no difference between TCP and UDP--is correct.
• "However, another concern I suppose is maybe I am sending update messages to frequently." Actually, when you compare the server logs to the client logs, you might discover that one side thinks it's sending lots of little packets frequently, but the other side actually receives big packets infrequently. If I were going to suggest something to try in the absence of data, I would say to make sure you have Nagling turned off, because it's on by default and you almost certainly do not want it.
• "Right now every .125 seconds" You might actually see your game perform better if you increased the update frequency. Measure, make a change, measure again!
• "<< "\r\n";" You should not change your serialization format in the absence of data. That having been said... yeah, once you have data, I think the odds are very good that data will tell you that your text serialization is way too verbose. Clients won't care; it's the poor server, who consumes 5-6x the bandwidth of any one client, who's getting pounded by this. Or rather who may be getting pounded by this--gather data to prove it!
• "Every bullet that is generate will send to the server the creation of the bullet," Okay, if you wanted to make just one change in the absence of any data, go ahead and fix that, just on principle. :) But I think your data will show that that's just a small part of the problem.
• "maybe my router's firewall is detecting the influx of packets... I don't think that's the case" I don't think that's the case either.

So: Go gather data and come back!