Sign in to follow this  
Oammar

How can I improve my network performance?

Recommended Posts

Today I gathered my group together in which we had 6 of us trying out the networked game play. It didn't go too well. When there is only 2 of us everything works as expected and it's enjoyable. However, some strange discrepancies started occurring when there were 6 of us; 3 or more really. Individuals started to time-out mainly, some were even crashing for some unknown reasons. Mainly tho, the time-outs is what I'm concerned about. That isn't supposed to happen unless they don't receive a reply from the server after 30 seconds or so, or if they disconnect manually.

 

One of the strange things I noticed during the test on the Server log was multiple requests for the particular individual to exit the game, which was very strange because they weren't requesting to do so.

 

My network system is currently just TCP, which I think may be the root of my problem. I've been thinking that all of my packets should be guaranteed to be delivered timely since we are only working across LAN, I'm starting to become doubtful. However, another concern I suppose is maybe I am sending update messages to frequently.

 

Right now every .125 seconds and if the player's transform changed I send a message that looks like this:

// send our transform information to the other players
  transMsg << NetworkManager::GetInstance().GetServerCmdFromMessageType(MESSAGE_TYPE::Game)
           << "!SPECIFICOBJECTMESSAGE " << netCompRef.GenerateMessageID("UPDATEOBJTRANSFORM")
           << gameObj->m_Pos.GetX()   << NLDS << gameObj->m_Pos.GetY()   << NLDS << gameObj->m_Pos.GetZ()   << NLDS
           << playerCam->m_Rot.GetX() << NLDS << playerCam->m_Rot.GetY() << NLDS << playerCam->m_Rot.GetZ() << NLDS
           << gameObj->m_Scale << NLDS << "\r\n";

I'm also doing something not good for when we are shooting. Every bullet that is generate will send to the server the creation of the bullet, which since we have rapid fire now is definitely sending a ton of messages to the server, then back to the other connected clients. Also, another thing I was thinking is maybe my router's firewall is detecting the influx of packets and is thinking it's a DDOS or something, which then terminates the connection on that end. However, I don't think that's the case, but it might be possible.

 

Thanks for any insight in helping me improve my networking, I'm willing to hear any suggestions.

Share this post


Link to post
Share on other sites

Today I gathered my group together in which we had 6 of us trying out the networked game play. It didn't go too well. When there is only 2 of us everything works as expected and it's enjoyable. However, some strange discrepancies started occurring when there were 6 of us; 3 or more really. Individuals started to time-out mainly, some were even crashing for some unknown reasons. Mainly tho, the time-outs is what I'm concerned about. That isn't supposed to happen unless they don't receive a reply from the server after 30 seconds or so, or if they disconnect manually.
 
One of the strange things I noticed during the test on the Server log was multiple requests for the particular individual to exit the game, which was very strange because they weren't requesting to do so.
 
My network system is currently just TCP, which I think may be the root of my problem. I've been thinking that all of my packets should be guaranteed to be delivered timely since we are only working across LAN, I'm starting to become doubtful. However, another concern I suppose is maybe I am sending update messages to frequently.
 
Right now every .125 seconds and if the player's transform changed I send a message that looks like this:

// send our transform information to the other players
  transMsg << NetworkManager::GetInstance().GetServerCmdFromMessageType(MESSAGE_TYPE::Game)
           << "!SPECIFICOBJECTMESSAGE " << netCompRef.GenerateMessageID("UPDATEOBJTRANSFORM")
           << gameObj->m_Pos.GetX()   << NLDS << gameObj->m_Pos.GetY()   << NLDS << gameObj->m_Pos.GetZ()   << NLDS
           << playerCam->m_Rot.GetX() << NLDS << playerCam->m_Rot.GetY() << NLDS << playerCam->m_Rot.GetZ() << NLDS
           << gameObj->m_Scale << NLDS << "\r\n";
I'm also doing something not good for when we are shooting. Every bullet that is generate will send to the server the creation of the bullet, which since we have rapid fire now is definitely sending a ton of messages to the server, then back to the other connected clients. Also, another thing I was thinking is maybe my router's firewall is detecting the influx of packets and is thinking it's a DDOS or something, which then terminates the connection on that end. However, I don't think that's the case, but it might be possible.
 
Thanks for any insight in helping me improve my networking, I'm willing to hear any suggestions.


are you sending a long string for each message ? if so you should switch to a binary format, a 32 bit float uses 4 bytes, a 32 bit float encoded as a string tends to use 7+ bytes, things like "!SPECIFICOBJECTMESSAGE " is 23 bytes (could probably be replaced by a 1 or 2 byte number), you also get a nice chunk of CPU overhead from the encoding/decoding (converting numeric values to strings and concatenating them is not cheap)

Your timeout / disconnect problems are probably caused by something else though, (on a LAN you should be able to send tons of data without any problems)

Share this post


Link to post
Share on other sites

I've been able to reproduce the random disconnect bug we were encountering earlier today. It seems whenever more than 2 people join, 3 people in my test case, one of the other clients will get dropped on the server end, but only after an arbitrary amount of time has passed. I was monitoring the time since last ping on my server end and that's never reaching the time-out period. Something else in my server code is doing something strange somewhere. I really like the idea of reducing the size of my sent packet, I am in the process of working on a way to register the particular network messages as enumerations, also to reduce my string comparisons. I hope that will help also.

Edited by Oammar

Share this post


Link to post
Share on other sites
I see you sending, but I don't see you parsing.
TCP does not guarantee that a single call to send() ends up in a single call to recv().
You can get more, or fewer, calls to recv() on the rececing end than you make to send() on the sending end.
Thus, all TCP reception needs to be into a buffer of some sort, and you parse a full message when you have a full message in the buffer.
There will also likely be additional data in the buffer after parsing the full message; save this data because it's the start of the next message (and may be the complete next message, too.)

So, step 1 in debugging would be to make sure that you treat run-together messages correctly.
Step 2 in debugging would be to read whatever switch() statements you have to make sure you didn't lose a "break" somewhere.
(That would easily cause unexpected commands to be decoded.)

Step 3 is to make it possible to record all incoming data on the server or on a client, and later play it back from scratch, to be able to debug whatever problems.
This is actually pretty easy to implement, and super, super valuable. In the end, you can ship this as a "game replay" feature for the client.
You would then capture a gameplay session, and put the data on your development machine. Put a breakpoint in "player requests exit" and Bob's your uncle.

Share this post


Link to post
Share on other sites

As hplus0603 and ProfBeeman have suggested the send/recv calls don't always align especially when network traffic picks up as you have suggested it does.

 

Some middleware libraries like 0MQ (ZeroMQ) wrap this for you and provide messaging patterns like you are looking for.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this