Sign in to follow this  

TCP Layer problems

This topic is 3703 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I have fully implemented a tcp layer that should of worked in a general network engine. As tcp is stream based, i am dividing the the received bytes into packets which are checked for correctness. this works... I tried a simple game where the server has around 40 network objects. the client can control 4 of these objects with his keyboard and mouse. the current implementation is: - client sends mouse +keyboard input - client receives update messages for each object - server has virtual keyboard+mouse per client - server runs all game logic - server transmits update object packets when there are 40 objects, the size of the pending data on the tcp layer is very large and most of the times new[] fails to allocate memory. I am having the server send all the network objects states every game loop. I re-tried the same test using UDP, it worked like a charm. Is it possible that TCP is this slow even on LAN? I would really like some experienced user opinion because I think I might be doing something wrong.

Share this post


Link to post
Share on other sites
I'm not sure how your implementation is setup but with UDP you're basically stateless, so it sounds like maybe you have some IO blocking issues that are slowing you down with your TCP setup? Not quite sure why you would have malloc errors though, I'll have to think about that one. Any code snippets you can post?

Share this post


Link to post
Share on other sites
Quote:
when there are 40 objects, the size of the pending data on the tcp layer is very large and most of the times new[] fails to allocate memory. I am having the server send all the network objects states every game loop.

I re-tried the same test using UDP, it worked like a charm.


Are the messages you send over UDP reliable?

Unlike TCP, which will recover missing traffic, UDP won't.

This would explain the congestion with TCP - there's too much pending data waiting for ack, resulting in saturation of network buffers. With UDP, if the buffer is full, incoming messages will be discarded.

My guess would be you're not processing fast enough for the traffic you're receiving, resulting in congestion.

I don't know why new[] would fail though. Network stack should have fixed buffers, and any traffic above that will be discarded. New failing would indicate other memory problems.

How much traffic do 40 objects generate? How often are you updating? While 100Mbit is more than enough for this size of world, your encoding might be saturating something. For example, sending every mouse move can generate 50-200+ messages per second. And while this isn't breaking anything, if your server isn't able to handle this at same rate, you will run into problems.

Share this post


Link to post
Share on other sites
Thanks for the info.
It seems with input updates and many clients in session the server using tcp will be very slow. If every object needs to synchronize this means 40 messages per object multiply that by the number of connected clients. and this is per second.
one solution is to have 1 update message for all the objects (compressed of course) and send that over tcp. the second solution (this is what i am doing right now). is to have game state server messages using udp (it is not that reliable) but with 60 message per second for every object it works fine.

Share this post


Link to post
Share on other sites
First: synchronization in network games is typically done at rates between 1 Hz (one update a second per object) and 30 Hz (thirty updates a second per object). Updates are not full state dumps, but instead only deltas, or maybe even only input needed to co-simulate the object on the client.

Second: Are you using blocking sockets? Are you using select()? How do you deal with partial writes to the TCP socket for updates, if the outgoing buffer is full?

Third: You must manage the outgoing bandwidth. That's something every networked game does. Select what to update, each time you update; don't send everything every time.

Fourth: If new[] fails, then you have a bug. Unless your objects have a state size of a megabyte each...

Fifth: Have you turned on the TCP_NODELAY option?

Share this post


Link to post
Share on other sites
Sorry if it took me a while to write back but i have been really busy.
Anyways here is the answers:
For a start I am doing a stress test, this means I am trying to put the worst case scenario to test the base network engine.
First: synchronization is done at 60hz, the server sends full object states.
Client sends full input states.
Of course they can be deltas but as i said i am trying to use as much bandwidth as possible in order to test the engine.

Second: I am using nonblocking sockets In comibation with another thread polling the sockets (using select). if there is data on a socket, if it is UDP read the data and enqueue. (the messages queue is managed on the main thread to minimize wait time on the Networking thread). If the socket is TCP, read the oncoming buffer. each packet has its header, if a packet is less than the header size this means it is an incomplete, so just read the remainder of the packet and continue

Third: Already answered in 1st.

Fourth: new has been fixed it was a tcp send/receive error, usually i m sending a structure which is the header followed by the data in the same message. the problem was if the tcp stream received was even less than the header size, this was fixed.

Fifth: TCP_NODELAY is on

I have also another layer implemented above this for managing the network in the game I.E: client side prediction, checking what to send and what to send, having a minimum updates to send from the server side.

My big problem is the following:
I am testing this scenario:
- The server sends 60 updates per second for each object.
- the client sends 60 input updates per second.
I am having 2 planes fly around and shoot. the bullets are dynamic network objects which are updated by the server (created, destroyed, moved). the client just interpolates. so at anytime i can have around 400-600 sprites managed by the server. This is a lot of messages. of course the simulation would be slow. But using udp everything is working, If i am using tcp for sending the updates, after a while the network thread will just lock I am still trying to debug the error but it is a it hard using pthread on windows.

Share this post


Link to post
Share on other sites
Quote:
if a packet is less than the header size this means it is an incomplete, so just read the remainder of the packet and continue


There is no guarantee that the remainder of the packet will be there yet. Someone who opened a connection to your socket, and sent a single byte, and then left it open, would lock up your server if you just "read the rest."

Share this post


Link to post
Share on other sites
if i understand what you are saying this can happen at any time ...
i read your implementation @ http://www.mindcontrol.org/~hplus/stream-messages.html
I do not think it solves the issue you discussed, any solution?
I am thinking of re-writing the tcp receive in a more general way (the tcp receive is actually written by another guy in the team, i am just patching in it up so if it happens that patching will become messier i guess i might have to re-write it).

Share this post


Link to post
Share on other sites
If you look at the code later in the article, it buffers data until it has enough to dequeue a header, which means it can tell how much data it really has to have in the buffer.

A working implementation of the same can be found in the Etwork library: http://www.mindcontrol.org/~hplus/etwork/

Share this post


Link to post
Share on other sites
oh thanks a lot, this really helped but i was concerned about this function:


if( gotHdr_ && buf_.size() >= packet_.h.size ) {
packet_.data = buf_.data();
return &packet_;
}



returning the packet, but the buf.poll() could of easily read more than the packet size. but this can be easily managed by getting the true size from the header.
I have already looked on etwork and i find that it does its job in a nice and really organized manner. I wish i was able to include any external library in this project than my choice would be to use yours. Thanks for your help.

Share this post


Link to post
Share on other sites
I am now using a mix of tcp and udp sockets on the host and server.
this means having a tcp socket listening for connection requests and communicating via slave sockets and a udp socket for udp messages (Send/recv).
I have read in many places on the forum that i could have the tcp and udp sockets on the same port. my problem is that when a client sends messages via udp to the server address, the tcp socket on the Client shows there is pending data and gets that data. I am do not know how this is possible unless i am missing some important option.

Share this post


Link to post
Share on other sites
Quote:
when a client sends messages via udp to the server address, the tcp socket on the Client shows there is pending data and gets that data


That's not possible. It has to be happening some other way. Are you using straight WinSock, or some library? No matter which, trace through the code, and see how you call socket(), to make sure you get two separate protocols. Also, when sending, make sure you send on the socket you think you're sending on.

Share this post


Link to post
Share on other sites
ignore that, it was caused by an engine bug and my lack of sleep. thanks for your help.
anyways everything seems running smoothly now.
What i want to add is that for tcp, i found it is faster to associate the socket with a buffer.
when there is data to be read (via select), then read all the pending data onto the buffer, then fragment that data into packets. if some data still remains and a packet needs more info to be constructed (i.e. the fragment size is less than the size of the header, or less than the packet's size), then the next check would read the remaining data and add it to the content of the buffer thus constructing the packet.



[Edited by - ZeRaW on October 26, 2007 9:25:45 AM]

Share this post


Link to post
Share on other sites

This topic is 3703 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this