Introduction

Whether you're interested in making an FPS or RTS, you've probably heard that you should use UDP. It's probably because of its speed. Using TCP with the TCP_NODELAY option (which means it doesn't wait for enough data to be buffered before sending) may not be enough as TCP also does congestion control and you may want to do voice chat, which is better done using a lossy protocol like UDP. TCP doesn't allow you to adjust the "sliding window", which means you might not reach the full speed of the communication channel if it has a high delay (search for "bandwidth-delay factor" for more information). If one packet doesn't arrive in TCP and is lost, TCP stops all traffic flow until it arrives, resulting in pauses. The packet header in TCP is also 20 bytes, as opposed to 6 bytes in UDP plus a few for reliability. Combining TCP and UDP is not an option as one induces packet loss in the other. So you should use UDP, but how do you guarantee packets are delivered and in the order they were sent in? Also, if you're making an RTS, how do you make sure that clients are running exactly the same simulation? A small change can cause a butterfly effect and be the difference between one player winning and losing. For this you need reliable UDP (RUDP) and lockstep. In addition, you probably want to use the latest client-server methodology, which allows the game to not be held back by the slowest player, as is done in the older peer-to-peer model. Along the way we'll cover parity bit checking to ensure packet correctness and integrity. We'll also cover LAN networking and setting up a matchmaking server.

This free article version covers only the UDP implementation. If you are interested in the full-breadth of the topic discussed above, you can purchase the complete 70-page PDF document on the GDNet Marketplace.

Library The library that I use for networking is SDL2_net. This is just abstraction of networking to allow us to deploy to different platforms like Windows, Linux, iPhone, and Android. If you want, you can use WinSock or the native networking functions of Linux. They're all pretty much the same, with a few differences in initialization and function names. But I would recommand SDL2_net as that is cross-platform. Watch for a possible article on setting up and compiling libraries for details on how to set up SDL2_net for use in your projects if you aren't able to do this yourself. By the way, if you're doing voice chat, try PortAudio for getting microphone input on Windows, Mac, and Linux. For transmitting you'd need Speex or the newer Opus speech codec for encoding the data. Packet Header So let's jump in. The way you will keep track of the order packets are sent in and also guarantee delivery is by using an ack or sequence number. We will make a header that will be at the beginning of each packet we send. We also need to know what kind of packet this is. struct PacketHeader { unsigned short type; unsigned short ack; }; To make sure the compiler packs the packet data as tightly as possible, to reduce network data usage, we can remove byte padding by putting this around all of our packet definitions. // byte-align structures #pragma pack(push, 1) //.. packets go here ?????????????????????????????? // default alignment #pragma pack(pop) By the way, they're not really packets; packets are what we use in TCP, but the subset in UDP is called a datagram. UDP stands for "user datagram protocol". But I call them packets. Control/Protocol Packets Let's define some control/protocol packets that will be part of our reliable UDP protocol. #define PACKET_NULL 0 #define PACKET_DISCONNECT 1 #define PACKET_CONNECT 2 #define PACKET_ACKNOWLEDGMENT 3 #define PACKET_NOCONN 4 #define PACKET_KEEPALIVE 5 #define PACKET_NACK 6 struct BasePacket { PacketHeader header; }; typedef BasePacket NoConnectionPacket; typedef BasePacket AckPacket; typedef BasePacket KeepAlivePacket; struct ConnectPacket { PacketHeader header; bool reconnect; unsigned short yourlastrecvack; unsigned short yournextrecvack; unsigned short yourlastsendack; }; struct DisconnectPacket { PacketHeader header; }; We will send a ConnectPacket to establish a new connection to a computer that is listening on a certain port. If you're behind a router and have several computers behind it, your router doesn't know which computer to forward an incoming connection to unless that computer has already sent data from that port outside. This is why a matchmaking server is needed, unless playing on LAN. More will be covered later. We will send a DisconnectPacket to end a connection. We need to send an AckPacket (acknowledgment) whenever we receive a packet type that is supposed to be reliable (that needs to arrive and be processed in order and cannot be lost). If the other side doesn't receive the ack, it will keep resending until it gets one or times out and assumes the connection to be lost. This is called "selective repeat" reliable UDP. Occasionally, we might not need to send anything for a long time but tell the other side to keep our connection alive, for example if we've connected to a matchmaking server and want to tell it to keep our game in the list. When this happens, we need to send a KeepAlivePacket. Almost all of these packets are reliable, as defined in the second paragraph on this page, except for the AckPacket and NoConnectionPacket. When we're first establishing a connection, we will use the ack number to set as the start of our sequence. When there's an interruption and the other side has dropped us due to timeout, but we still have a connection to them, they will send a NoConnectionPacket, which is not reliable, but is sent every time we receive a packet from an unknown source (whose address we don't recognize as having established a connection to). Whenever this happens, we have to do a full reconnect as the sequence/ack numbers can't be recovered. This is important because we will have a list of packets that we didn't get a reply to that we will resend that need to be acknowledged. You will understand more as we talk about how ack/sequence numbers work. Lastly, the AckPacket's ack member is used to tell the other side which packet we're acknowledging. So an ack packet is not reliable, and is sent on an as-needed basis, when we receive a packet from the other side. User Control/Protocol Packets The rest of the packet types are user-defined (that's you) and depend on the type of application needed. If you're making a strategy game, you'll have a packet type to place a building, or to order units around. But there are also some control/protocol packets needed that are not part of the core protocol. These are packets for joining game sessions, getting game host information, getting a list of game hosts, informing a game client that the game room is full or that his version is older. Here are some suggested packet types for a multiplayer RTS or any kind of strategy game, but may also apply to FPS. These are used after a connection has been established. #define PACKET_JOIN 7 #define PACKET_ADDSV 8 #define PACKET_ADDEDSV 9 #define PACKET_GETSVLIST 10 #define PACKET_SVADDR 11 #define PACKET_SVINFO 12 #define PACKET_GETSVINFO 13 #define PACKET_SENDNEXTHOST 14 #define PACKET_NOMOREHOSTS 15 #define PACKET_ADDCLIENT 16 #define PACKET_SELFCLIENT 17 #define PACKET_SETCLNAME 18 #define PACKET_CLIENTLEFT 19 #define PACKET_CLIENTROLE 20 #define PACKET_DONEJOIN 21 #define PACKET_TOOMANYCL 22 #define PACKET_MAPCHANGE 23 #define PACKET_CLDISCONNECTED 24 #define PACKET_CLSTATE 25 #define PACKET_CHAT 26 #define PACKET_MAPSTART 27 #define PACKET_GAMESTARTED 28 #define PACKET_WRONGVERSION 29 #define PACKET_LANCALL 30 #define PACKET_LANANSWER 31 Lockstep Control Packets Once a game has been joined and started, RTS games also need to control the simulation to keep it in sync on all sides. A class of packet types that control the lockstep protocol we will call the lockstep control packets. #define PACKET_NETTURN 29 #define PACKET_DONETURN 30 User Command / In-Game Packets Any user command or input that effects the simulation needs to be bundled up inside a NetTurnPacket by the host and sent to all the clients as part of lockstep. But first it is sent to the host on its own. More on this later. Here are some example packets I use. #define PACKET_PLACEBL 32 #define PACKET_CHVAL 33 #define PACKET_ORDERMAN 34 #define PACKET_MOVEORDER 35 #define PACKET_PLACECD 36 Initialization You should really read some C/C++ UDP or SDL2_net UDP example code before you try to proceed with this, and the basics of that are not what I'm covering here. But nevertheless, I will mention that you need to initialize before you use SDL2_net or WinSock. if(SDLNet_Init() == -1) { char msg[1280]; sprintf(msg, "SDLNet_Init: %s\n", SDLNet_GetError()); SDL_ShowSimpleMessageBox(SDL_MESSAGEBOX_ERROR, "Error", msg, NULL); } Here are some examples for low-level (not using SDL) UDP networking: https://www.cs.rutgers.edu/~pxk/417/notes/sockets/udp.html http://www.binarytides.com/programming-udp-sockets-c-linux/ http://www.codeproject.com/Articles/11740/A-simple-UDP-time-server-and-client-for-beginners They all cover the same material. You just need to know the basics of how to intialize, send data, and receive data using UDP (not TCP). Net Update Loop So somewhere in your frame loop, where you update your game state and render everything, you will add a call to UpdNet, which will process received packets. The skeleton of it will look roughly like this. //Net input void UpdNet() { int bytes; UDPpacket *in; UDPsocket* sock = &g_sock; if(!sock) return; in = SDLNet_AllocPacket(65535); do { in->data[0] = 0; bytes = SDLNet_UDP_Recv(*sock, in); IPaddress ip; memcpy(&ip, &in->address, sizeof(IPaddress)); if(bytes > 0) TranslatePacket((char*)in->data, bytes, true, &g_sock, &ip); } while(bytes > 0); SDLNet_FreePacket(in); } Following, inside the do-while loop, is the equivalent code in regular Berkeley sockets using recvfrom, in case you're not using SDL2_net: //Net input void UpdNet() { int bytes; int* sock = &g_sock; if(!sock) return; do { struct sockaddr_in from; socklen_t fromlen = sizeof(struct sockaddr_in); char buffer[65535]; bytes = recvfrom(g_socket, buffer, 65535, 0, (struct addr *)&from, &fromlen); if(bytes > 0) TranslatePacket(buffer, bytes, true, &g_sock, &ip); } while(bytes > 0); } This basically loops while we still have data to process in the buffer. If we do, we send it to the TranslatePacket function, which takes as parameters: the buffer of data ("buffer") the number of bytes received ("bytes") whether we want to check the acknowledgement/sequence number and process it in order ("checkprev"), as we might choose to send a dummy packet that we stored to reuse our command switch functionality for lockstep batch packets, or whether we want to process it right away the socket ("sock") and the IP address and port it came from ("from"). void TranslatePacket(char* buffer, int bytes, bool checkprev, UDPsocket* sock, IPaddress* from) We will get to TranslatePacket next, but first take a look at these function calls we will also have at the end of the UpdNet function. KeepAlive(); CheckConns(); ResendPacks(); #ifndef MATCHMAKER CheckAddSv(); CheckGetSvs(); #else SendSvs(); #endif KeepAlive() tries to keep connections alive that are about to time out. CheckConns() checks for connections that we've closed or that have timed out from unresponsiveness and recycles them. ResendPacks() tries to resend packets that we haven't received an acknowledgement for, once they've waited long enough. Then we have a preprocessor check to see whether we're compiling for the matchmaking server or the client program/app. If we're a game client or host, we only care about CheckAddSv() and CheckGetSv(). CheckAddSv() checks whether our game host address is up on the matchmaker list and getting the information on each one of the received hosts. It also removes hosts from our list that we have lost a connection too (due to time out). CheckGetSv() makes sure the matchmaker will send us the next host from its list if we've requested to get a host list. If we're the matchmaker however, we only care about SendSvs(), which sends the next host in the list for each requesting client when the last one has been ack'd. Connection Class Because there's no concept of a connection in UDP, we need to make it ourselves. class NetConn { public: unsigned short nextsendack; unsigned short lastrecvack; bool handshook; IPaddress addr; //TODO change these to flags bool isclient; //is this a hosted game's client? or for MATCHMAKER, is this somebody requesting sv list? bool isourhost; //is this the currently joined game's host? cannot be a host from a server list or something. for MATCHMAKER, it can be a host getting added to sv list. bool ismatch; //matchmaker? bool ishostinfo; //is this a host we're just getting info from for our sv list? //bool isunresponsive; unsigned long long lastsent; unsigned long long lastrecv; short client; float ping; bool closed; bool disconnecting; void expirein(int millis); #ifdef MATCHMAKER int svlistoff; //offset in server list, sending a few at a time SendSvInfo svinfo; #endif //void (*chcallback)(NetConn* nc, bool success); //connection state change callback - did we connect successfully or time out? NetConn() { client = -1; handshook = false; nextsendack = 0; //important - reply ConnectPacket with ack=0 will be //ignored as copy (even though it is original) if new NetConn's lastrecvack=0. lastrecvack = USHRT_MAX; isclient = false; isourhost = false; ismatch = false; ishostinfo = false; //isunresponsive = false; lastrecv = GetTicks(); lastsent = GetTicks(); //chcallback = NULL; #ifdef MATCHMAKER svlistoff = -1; #endif ping = 1; closed = false; } }; "nextsendack" is the outgoing sequence number that the next reliable packet will have. We increment it by one each time and it wraps around from 0 when it maxes out. "lastrecvack" is the inbound sequence number of the last received reliable packet (the next one will be greater until it wraps around). Because we can send and receive independently, we keep two acks/sequences. When we start a connection, "nextsendack" is 0 (for the first packet sent, the ConnectPacket), and "lastrecvack" is 65535, which is the maximum unsigned short value, before it wraps around to 0. Although nextsendack can be set to anything (and should) as long as we set the ack of the first ConnectPacket to that, and that is more secure, as it is harder to predict or crack and probably protects data better (but I haven't tried it). "handshook" tells us whether the other side has acknowledged the ConnectPacket (and therefore created an accompanying NetConn connection instance for us). When we receive a ConnectPacket, we set "handshook" to true and acknowledge them, recording the inbound ack and setting "nextsendack" to 0. If "handshook" is true, it tells us that we can now send reliable packets on the connection to this address, as the sequence numbers are in place. IPaddress is the SDL2_net IP address and port structure, the equivalent of sockaddr_in in regular Berkeley sockets. Translating Packets This is what TranslatePacket does. When we "translate" or process the packet we want to know if it's an old packet that we've already processed once or if it's ahead of the next expected packet, if it's meant to be reliable as defined previously. We also acknowledge any packets that are reliable. And then finally we execute them. The game host does an extra part in beginning of the function, checking if any client connections are unresponsive or have become responsive again and relays that information to other clients, so the blame can be pinned on the lagger. The TranslatePacket() function has this basic outline: 1. Match address to a connection 2. Update last received timestamp if match found 3. Check packet type if we need to check sequence number or if we process it right away 4. Check sequence number for one of three cases (behind, current, or future) 5. Acknowledge packet if needed 6. If we don't recognize the connection and it's supposed to be reliable, tell the other side that we don't have a connection with them 7. Execute the packet 8. Execute any buffered packets after the current one (in order) 9. And update the last received sequence number to the last packet executed Step by step, the function is: void TranslatePacket(char* buffer, int bytes, bool checkprev, UDPsocket* sock, IPaddress* from) { //1. Match address to a connection PacketHeader* header = (PacketHeader*)buffer; NetConn* nc = Match(from); We pass an IPaddress struct pointer to Match which returns the matching connection or NULL on failure. Then, if we got a match, we update the last received time for the connection. If the connection is associated with client in the game room, and that client was previously unresponsive, we can mark it as responsive again and tell the other clients. //If we recognize this connection... if(nc) { //2. Update the timestamp of the last received packet nc->lastrecv = GetTicks(); #ifndef MATCHMAKER //check if was previously unresponsive //and if (s)he was, tell others that (s)he //is now responsive. if(nc->client >= 0) { Client* c = &g_client[nc->client]; //was this client unresponsive? if(c->unresp) { //it's now responsive again. c->unresp = false; //if we're the game host if(g_netmode == NETM_HOST) { //inform others ClStatePacket csp; csp.header.type = PACKET_CLSTATE; csp.chtype = CLCH_RESP; csp.client = nc->client; //send to all except the original client (nc->addr) SendAll((char*)&csp, sizeof(ClStatePacket), true, false, &nc->addr); } } } #endif } We know that certain packets are meant to be processed right away, without checking for them to be processed in sequence. For example, acknowedgement packets are non-reliable and don't need to be processed in a specific order. Connect packets are to be executed as soon as they are received, because no other packet is supposed to be sent with them. Same for disconnect packets, "no connection" packets, LAN call, and LAN answer. //3. Check packet type if we need to check sequence number or if we process it right away //control packets //don't check sequence for these ones and process them straight away //but acknowledge CONNECT and DISCONNECT switch(header->type) { case PACKET_ACKNOWLEDGMENT: case PACKET_CONNECT: //need to send back ack case PACKET_DISCONNECT: //need to send back ack case PACKET_NOCONN: case PACKET_NACK: case PACKET_LANCALL: case PACKET_LANANSWER: checkprev = false; break; default: break; } If it's not one of those packet types, checkprev=true, and we check the sequence number. These are reliable packets that must be processed in the order they were sent in. If we're missing a packet in the sequence, we will buffer the packets after it while we wait for the missing packet to arrive. "next" will be the next expected sequence number (the one after lastrecvack in NetConn). "last" will be updated each time we execute a packet, to update lastrecvack with the last one. unsigned short next; //next expected packet ack unsigned short last = PrevAck(header->ack); //last packet ack to be executed //4. Check sequence number for one of three cases (behind, current, or future) //If checkprev was set (directly above), we need to check the sequence. //It must be a recognized NetConn; otherwise we don't have any sequence numbers. if(checkprev && nc != NULL) { // ?????????????????????????????? check sequence number (check snippet further down) ... } Then we acknowledge the packet if it's meant to be reliable. Acknowledgement packets don't need acknowledgements themselves. A "no connection" packet tells us the other side doesn't even have sequence numbers for us, so there's no point acknowledging it. Usually, if checkprev=false, we don't check the packet sequence so we don't care about acknowledging it, but for connect and disconnect packets we must acknowledge because the other side expects a success signal back. //5. Acknowledge packet if needed procpack: //We might disconnect further down in PacketSwitch() //So acknowledge packets while we still have the sequence numbers nc = Match(from); //Don't acknowledge NoConn packets as they are non-reliable, //and ack'ing them would cause a non-ending ack loop. if(header->type != PACKET_ACKNOWLEDGMENT && header->type != PACKET_NOCONN && sock && nc) { Acknowledge(header->ack, nc, from, sock, buffer, bytes); } //Always acknowledge ConnectPacket's else if( header->type == PACKET_CONNECT && sock ) { Acknowledge(header->ack, NULL, from, sock, buffer, bytes); } //And acknowledge DisconnectPacket's else if(header->type == PACKET_DISCONNECT && sock) { Acknowledge(header->ack, NULL, from, sock, buffer, bytes); } If we got disconnected from the other side and for some reason they retained the connection, we'll get packets that we have to tell the other side we can't process. They can then show an error to the user or try to reconnect. //6. If we don't recognize the connection and it's supposed to be reliable, tell the other side that we don't have a connection with them //We're getting an anonymous packet. //Maybe we've timed out and they still have a connection. //Tell them we don't have a connection. //We check if sock is set to make sure this isn't a local //command packet being executed. if(!nc && header->type != PACKET_CONNECT && header->type != PACKET_NOCONN && header->type != PACKET_LANCALL && header->type != PACKET_LANANSWER && sock) { NoConnectionPacket ncp; ncp.header.type = PACKET_NOCONN; SendData((char*)&ncp, sizeof(NoConnectionPacket), from, false, true, NULL, &g_sock, 0, NULL); return; } Then we execute packets. First, any packets before the current received one. Then the one we just received. And then any that we buffered that come after it. The reason we execute packets that came BEFORE is because we may have a case like this: packet 1 received packet 2 received packet 5 received We'll be able to execute packets 1 and 2 even though the current is 5. updinack: //7. Execute the packet //8. Execute any buffered packets after the current one (in order) // Translate in order if(checkprev && nc) { last = PrevAck(header->ack); last = ParseRecieved(next, last, nc); } // Translate in order if(NextAck(last) == header->ack || !checkprev) { PacketSwitch(header->type, buffer, bytes, nc, from, sock); last = header->ack; } // Translate in order if(checkprev && nc && last == header->ack) { while(true) { if(!Recieved(last+1, last+1, nc)) break; last++; ParseRecieved(last, last, nc); } } Finally, we update the received sequence number. We have to match up the connection pointer with the address again, because the instance it was pointing to might have been erased, or it might have appeared when it wasn't previously, as a connection is erased or created respectively. For non-reliable packets we don't update the sequence number. For connect or disconnect packets, we only set the sequence number inside the PacketSwitch read function call when we create a connection. //9. And update the last received sequence number to the last packet executed //have to do this again because PacketSwitch might //read a ConnectPacket, which adds new connections. //also connection might have //been Disconnected(); and erased. nc = Match(from); //ack Connect packets after new NetConn added... //Don't acknowledge NoConn packets as they are non-reliable if(header->type != PACKET_ACKNOWLEDGMENT && header->type != PACKET_NOCONN && sock && nc && checkprev) { if(header->type != PACKET_CONNECT && header->type != PACKET_DISCONNECT) nc->lastrecvack = last; } } The PacketSwitch() at the end is what executes the packet. It might better be called ExecPacket(). The Match() function at the top compares the "addr" port and IP address integers to every known connection and returns the match, or NULL on failure. NetConn* Match(IPaddress* addr) { if(!addr) return NULL; for(auto ci=g_conn.begin(); ci!=g_conn.end(); ci++) if(Same(&ci->addr, addr)) return &*ci; return NULL; } bool Same(IPaddress* a, IPaddress* b) { if(a->host != b->host) return false; if(a->port != b->port) return false; return true; } The packet is "old" if we've already buffered it (but it's ahead of the next expected ack/sequence number that we processed), or if its ack/sequence is behind our connection class's "lastrecvack". We use an unsigned short for the sequence number, which holds a maximum value of 65535. Because we might exceed this value after 36 minutes if we send 30 packets a second, we wrap around and thus, there's a "sliding window" of values that are considered to be in the past (don't confuse this with the "sliding window" packet range that might be being reliably resent at any given moment). We can check if an ack is in the past (behind what is already executed) using PastAck(): bool PastAck(unsigned short test, unsigned short current) { return ((current >= test) && (current - test <= USHRT_MAX/2)) || ((test > current) && (test - current > USHRT_MAX/2)); } Where PastAck tests whether "test" is behind or at "current". Let's look in more detail at the part in the middle of TranslatePacket that checks the sequence number. We define some variables. "next" will hold the current expected ack (lastrecvack+1) inside the following code block. "last" will hold the last packet to have been executed. For now, it's set to something, but it doesn't matter, as we update it at the end. unsigned short next; //next expected packet ack unsigned short last = PrevAck(header->ack); //last packet ack to be executed We only check the sequence numbers if it's a packet that makes checkprev=true and if it's from a recognized connection. if(checkprev && nc != NULL) { We set the "next" expected packet number. next = NextAck(nc->lastrecvack); //next expected packet ack last = next; //last packet ack to be executed Next, we check how the received packet's sequence number compares to the next expected one. //CASE #1: ???????????????old??????????????? packet if(PastAck(header->ack, nc->lastrecvack) || Recieved(header->ack, header->ack, nc)) { Acknowledge(header->ack, nc, from, sock, buffer, bytes); return; } //CASE #2: current packet (the next expected packet) if(header->ack == next) { // Translate packet last = next; } //CASE #3: an unbuffered, future packet else // More than +1 after lastrecvack? { /* last will be updated to the last executed packet at the end. for now it will hold the last buffered packet to be executed. */ unsigned short checklast = PrevAck(header->ack); if(Recieved(next, checklast, nc)) { // Translate in order last = checklast; goto procpack; } else { AddRecieved(buffer, bytes, nc); if(Recieved(next, checklast, nc)) { // Translate in order last = checklast; goto procpack; } else { //TODO //how to find which ack was missed, have to go through all buffered //this is something somebody smart can do in the future //NAckPacket nap; //nap.header.type = PACKET_NACK; //nap.header.ack = } } } } As can be seen, there are three possible cases for the inbound packet's sequence number: it is either, 1.) behind or buffered, 2.) current expected, or 3.) future unbuffered. Case 1: behind and buffered received packets If we've already dealt with (executed) the packet, we simply acknowledge it again and return from TranslatePacket() with no further action. if(PastAck(header->ack, nc->lastrecvack) || Recieved(header->ack, header->ack, nc)) { Acknowledge(header->ack, nc, from, sock, buffer, bytes); return; } In the the second testcase of the if statement (packet is buffered received), we check if we've already buffered it, using Recieved(): //check when we've recieved a packet range [first,last] inclusive bool Recieved(unsigned short first, unsigned short last, NetConn* nc) { OldPacket* p; PacketHeader* header; unsigned short current = first; unsigned short afterlast = NextAck(last); bool missed; //go through all the received packets and check if we have the complete range [first,last] do { //for each number in the sequence... missed = true; //look through each packet from that address for(auto i=g_recv.begin(); i!=g_recv.end(); i++) { p = &*i; header = (PacketHeader*)p->buffer; //is this the sequence number we're looking for? if(header->ack != current) continue; //is this the correct address? if(!Same(&p->addr, &nc->addr)) continue; //go to next number in the sequence now that we know we have the previous one current = NextAck(current); missed = false; break; } //if we finished the inner loop and ???????????????missed??????????????? is still false, we missed a number in the sequence, so return false if(missed && current != afterlast) return false; //continue looping until we've arrived at the number after the ???????????????last??????????????? number } while(current != afterlast); //if we got here, we got all the numbers return true; } "g_recv" is a linked list of OldPacket's. We go through each sequence number between "first" and "last" and check if we have each and every one. Because we use the received packet's ack number for both parameters in Case 1, we only check if we've buffered that one packet. Because g_recv holds inbound packets from every address we're connected to, we have to check to match the address when comparing ack numbers. You can store g_recv in the NetConn's and this might be more efficient. Buffered Packets The OldPacket class holds the byte array for the packet and the address and port of the sender (or the outbound port and address for outgoing buffered packets). class OldPacket { public: char* buffer; int len; unsigned long long last; //last time resent unsigned long long first; //first time sent bool expires; bool acked; //used for outgoing packets //sender/reciever IPaddress addr; void (*onackfunc)(OldPacket* op, NetConn* nc); void freemem() { if(len <= 0) return; if(buffer != NULL) delete [] buffer; buffer = NULL; } OldPacket() { len = 0; buffer = NULL; onackfunc = NULL; acked = false; } ~OldPacket() { freemem(); } OldPacket(const OldPacket& original) { len = 0; buffer = NULL; *this = original; } OldPacket& operator=(const OldPacket &original) { freemem(); if(original.buffer && original.len > 0) { len = original.len; if(len > 0) { buffer = new char[len]; memcpy((void*)buffer, (void*)original.buffer, len); } last = original.last; first = original.first; expires = original.expires; acked = original.acked; addr = original.addr; onackfunc = original.onackfunc; } else { buffer = NULL; len = 0; onackfunc = NULL; } return *this; } }; It has some extra fields for outbound packets. Case 2: current expected received packets The second case is when the received packet is the next expected one, which means we received it in the correct order without repeats. The next expected (current) packet is the one after the "last received" one (lastrecvack). The variable "next" here will hold that ack. It is equal to nc->lastrecvack + 1, so you can use that instead of the function "NextAck". next = NextAck(nc->lastrecvack); //next expected packet ack last = next; //last packet ack to be executed //CASE #2: current packet (the next expected packet) if(header->ack == next) { // Translate packet last = next; } If it matches "next" we will process the packet and acknowledge it further down. We record the "last" packet executed, to update the sequence number. Case #3: future, unbuffered received packets If we reach "else" it means we have an unbuffered, future packet. //CASE #3: an unbuffered, future packet else // More than +1 after lastrecvack? { /* last will be updated to the last executed packet at the end. for now it will hold the last buffered packet to be executed. */ unsigned short checklast = PrevAck(header->ack); if(Recieved(next, checklast, nc)) { // Translate in order last = checklast; goto procpack; } else { AddRecieved(buffer, bytes, nc); if(Recieved(next, checklast, nc)) { // Translate in order last = checklast; goto procpack; } else { //TODO //how to find which ack was missed, have to go through all buffered //this is something somebody smart can do in the future //NAckPacket nap; //nap.header.type = PACKET_NACK; //nap.header.ack = } } } We check if we have a range of buffered packets up to this one. If we have a complete range, starting from the current (expected next) packet, we can execute them (because we only run them in the order they're sent in) and increase lastrecvack to equal "last". We move up lastrecvack at the end of TranslatePacket. We might have more buffered packets after the received one. That is why we check for any extra packets and store the last executed one's ack number in "last". If we don't have a complete set of packets up to the received one, we call AddRecieved (buffer it). void AddRecieved(char* buffer, int len, NetConn* nc) { OldPacket* p; g_recv.push_back(OldPacket()); p = &*g_recv.rbegin(); p->freemem(); p->addr = nc->addr; p->buffer = new char[ len ]; p->len = len; memcpy((void*)p->buffer, (void*)buffer, len); memcpy((void*)&p->addr, (void*)&nc->addr, sizeof(IPaddress)); g_recv.push_back(p); } If we have to buffer it, it means it's ahead of the last executed packet, and there's one missing before it. If we wanted to only send selective repeats every second or so (if that was the delay on the channel and we didn't want to send some three copies of it before we received back an ack, and we're sure that loss of packets is minimal, and we'd rather leave the "sliding window" huge), we could use NAck's (negative ack's) to tell us when we've missed a packet. But selective repeat works pretty well. (Using nacks is a different kind of reliable UDP implementation.) Acknowledgements Further on we send ack's. void Acknowledge(unsigned short ack, NetConn* nc, IPaddress* addr, UDPsocket* sock, char* buffer, int bytes) { AckPacket p; p.header.type = PACKET_ACKNOWLEDGMENT; p.header.ack = ack; SendData((char*)&p, sizeof(AckPacket), addr, false, true, nc, sock, 0, NULL); } We use a SendData function for our RUDP implementation, shown and explained further down. Whenever we send data, we have to fill out a packet struct for that type of packet. At minimum, we have to set header.type so that the received end can know what packet type it is from reading the first 2 bytes of the packet. Executing Packet and Updating Sequence Number If we get to this point in TranslatePacket, we'll execute the packets in order. If we checked sequence numbers, and we have a connection, we'll execute the buffered previous packets, then the current received packets, then check for any future buffered packets. If we don't check the sequence, or don't have a connection, we just execute the one packet we received. updinack: // Translate in order if(checkprev && nc) { last = header->ack; last = ParseRecieved(next, last, nc); } // Translate in order if(NextAck(last) == header->ack || !checkprev) { PacketSwitch(header->type, buffer, bytes, nc, from, sock); last = header->ack; } // Translate in order if(checkprev && nc && last == header->ack) { while(true) { if(!Recieved(last+1, last+1, nc)) break; last++; ParseRecieved(last, last, nc); } } //have to do this again because PacketSwitch might //read a ConnectPacket, which adds new connections. //but also the connection might have //been Disconnected(); and erased. nc = Match(from); //ack Connect packets after new NetConn added... //Don't acknowledge NoConn packets as they are non-reliable if(header->type != PACKET_ACKNOWLEDGMENT && header->type != PACKET_NOCONN && sock && nc && checkprev) { if(header->type != PACKET_CONNECT && header->type != PACKET_DISCONNECT) nc->lastrecvack = last; } At the end we update the connection's "lastrecvack" to "last" one executed. If it's a ConnectPacket, we set the lastrecvack when reading the packet. Executing a buffered packet range We need to execute a packet range when we know we've got a complete sequence up to a certain ack. We return the last executed packet number here, in case it's behind "last". unsigned short ParseRecieved(unsigned short first, unsigned short last, NetConn* nc) { OldPacket* p; PacketHeader* header; unsigned short current = first; unsigned short afterlast = NextAck(last); do { bool execd = false; for(auto i=g_recv.begin(); i!=g_recv.end(); i++) { p = &*i; header = (PacketHeader*)p->buffer; if(header->ack != current) continue; if(!Same(&p->addr, &nc->addr)) continue; PacketSwitch(header->type, p->buffer, p->len, nc, &p->addr, &g_sock); execd = true; current = NextAck(current); i = g_recv.erase(i); break; } if(execd) continue; break; } while(current != afterlast); return PrevAck(current); } SendData We send data like so, passing the data bytes, size, address, whether it is meant to be reliable, whether we want it to expire after a certain time of resending (like a ConnectPacket that needs to fail sooner than the default timeout), the NetConn connection (which musn't be NULL if we're sending a reliable packet), the socket, the millisecond delay if we want to queue it to send a few moments from now, and a callback function to be called when it's acknowledged so we can take further action (like setting "handshook" to true for ConnectPacket's, or destroying the NetConn when a DisconnectPacket is acknowledged). void SendData(char* data, int size, IPaddress * paddr, bool reliable, bool expires, NetConn* nc, UDPsocket* sock, int msdelay, void (*onackfunc)(OldPacket* p, NetConn* nc)) { //is this packet supposed to be reliable? if(reliable) { //if so, set the ack number ((PacketHeader*)data)->ack = nc->nextsendack; //and add an OldPacket to the g_outgo list OldPacket* p; g_outgo.push_back(OldPacket()); p = &*g_outgo.rbegin(); p->freemem(); p->buffer = new char[ size ]; p->len = size; memcpy(p->buffer, data, size); memcpy((void*)&p->addr, (void*)paddr, sizeof(IPaddress)); //in msdelay milliseconds, p.last will be RESEND_DELAY millisecs behind GetTicks() p->last = GetTicks() + msdelay - RESEND_DELAY; p->first = p->last; p->expires = expires; p->onackfunc = onackfunc; //update outbound ack for this connection nc->nextsendack = NextAck(nc->nextsendack); } if(reliable && msdelay > 0) return; PacketHeader* ph = (PacketHeader*)data; if(reliable && (!nc || !nc->handshook) && (ph->type != PACKET_CONNECT && ph->type != PACKET_DISCONNECT && ph->type != PACKET_ACKNOWLEDGMENT && ph->type != PACKET_NOCONN) ) { Connect(paddr, false, false, false, false); return; } memcpy(out->data, data, size); out->len = size; out->data[size] = 0; SDLNet_UDP_Unbind(*sock, 0); if(SDLNet_UDP_Bind(*sock, 0, (const IPaddress*)paddr) == -1) { char msg[1280]; sprintf(msg, "SDLNet_UDP_Bind: %s\n",SDLNet_GetError()); ErrMess("Error", msg); //printf("SDLNet_UDP_Bind: %s\n",SDLNet_GetError()); //exit(7); } //sendto(g_socket, data, size, 0, (struct addr *)paddr, sizeof(struct sockaddr_in)); SDLNet_UDP_Send(*sock, 0, out); g_transmitted += size; SDLNet_FreePacket(out); } If it's reliable, we add an entry to the outbound OldPacket list. We set the "last" member variable of the OldPacket entry such that it is resent in a certain amount of time depending on when we delayed it to and the usual resend delay. If it's reliable and the delay is greater than 0, we don't take any action in this function after buffering it in the outbound list because we will send it after ResendPacks() is called. If it's reliable and we don't have a connection specified, we call Connect() to connect first, and return. It is also called if the connection hasn't finished the handshake (in which case Connect() will check to make sure that we have an outgoing ConnectPacket). The only case in which we don't need a handshook connection and send reliably is if we're sending a ConnectPacket or DisconnectPacket. The SendData function is called itself with "reliable" set to false when resending a reliable packet from a buffered outbound OldPacket container. The SendData function automatically sets the outbound ack for the reliable packets. Keeping Connections Alive As mentioned, there are three more functions in the UpdNet loop function: KeepAlive(); CheckConns(); ResendPacks(); The KeepAlive() function sends KeepAlive packets to connections that are expiring. It prevents the other side from closing the connection, and also triggers an ack packet back, preventing from the connection being closed locally. The default is to keep connections alive until the user decides to Disconnect them. //keep expiring connections alive (try to) void KeepAlive() { unsigned long long nowt = GetTicks(); auto ci = g_conn.begin(); //loop while we still have more connections to process... while(g_conn.size() > 0 && ci != g_conn.end()) { //if we haven't received a handshake back, or if it's closed, we don't need to be keep it alive if(!ci->handshook || ci->closed) { ci++; continue; } //otherwise, if it's reached a certain percent of the timeout period, send a KeepAlivePacket... if(nowt - ci->lastrecv > NETCONN_TIMEOUT/4) { //check if we're already trying to send a packet to get a reply bool outgoing = false; //check all outgoing packets for a packet to this address for(auto pi=g_outgo.begin(); pi!=g_outgo.end(); pi++) { //if(memcmp(&pi->addr, &ci->addr, sizeof(IPaddress)) != 0) if(!Same(&pi->addr, &ci->addr)) { continue; } outgoing = true; break; } //if we have an outgoing packet, we don't have to send a KeepAlivePacket if(outgoing) { ci++; continue; } //otherwise, send a KeepAlivePacket... KeepAlivePacket kap; kap.header.type = PACKET_KEEPALIVE; SendData((char*)&kap, sizeof(KeepAlivePacket), &ci->addr, true, false, &*ci, &g_sock, 0, NULL); } //check next connection next ci++; } } GetTicks() is our 64-bit timestamp function in milliseconds: unsigned long long GetTicks() { #ifdef PLATFORM_WIN SYSTEMTIME st; GetSystemTime (&st); _FILETIME ft; SystemTimeToFileTime(&st, &ft); //convert from 100-nanosecond intervals to milliseconds return (*(unsigned long long*)&ft)/(10*1000); #else struct timeval tv; gettimeofday(&tv, NULL); return (unsigned long long)(tv.tv_sec) * 1000 + (unsigned long long)(tv.tv_usec) / 1000; #endif } Checking and Pruning Connections Two more functions in UpdNet: CheckConns(); ResendPacks(); In CheckConns we do several things: 1. Send out periodic pings for all the players in the room for all the clients using Cl(ient)StatePacket's 2. Handle and close any connections that are not yet closed but have timed out because the last received message has been longer than NETCONN_TIMEOUT milliseconds ago 3. For closed connections, flush any buffered inbound or outbound OldPacket's, and erase the NetConn from the list 4. For unresponsive clients, inform other players of the lagger void CheckConns() { unsigned long long now = GetTicks(); // If we're not compiling for the matchmaker (the game app itself) #ifndef MATCHMAKER static unsigned long long pingsend = GetTicks(); //send out client pings if(g_netmode == NETM_HOST && now - pingsend > (NETCONN_UNRESP/2) ) { pingsend = now; for(int i=0; i; if(!c->on) continue; if(i == g_localC) continue; //clients will have their own ping for the host NetConn* nc = c->nc; if(!nc) continue; ClStatePacket csp; csp.header.type = PACKET_CLSTATE; csp.chtype = CLCH_PING; csp.ping = nc->ping; csp.client = i; SendAll((char*)&csp, sizeof(ClStatePacket), true, false, NULL); } } #endif auto ci = g_conn.begin(); while(g_conn.size() > 0 && ci != g_conn.end()) { //get rid of timed out connections if(!ci->closed && now - ci->lastrecv > NETCONN_TIMEOUT) { //TO DO any special condition handling, inform user about sv timeout, etc. #ifndef MATCHMAKER if(ci->ismatch) { g_sentsvinfo = false; } else if(ci->isourhost) { EndSess(); RichText mess = RichText("ERROR: Connection to host timed out."); Mess(&mess); } else if(ci->ishostinfo) ; //ErrMess("Error", "Connection to prospective game host timed out."); else if(ci->isclient) { //ErrMess("Error", "Connection to client timed out."); /* TODO combine ClDisconnectedPacket and ClientLeftPacket. use params to specify conditions of leaving: - of own accord - timed out - kicked by host */ //TODO inform other clients? ClDisconnectedPacket cdp; cdp.header.type = PACKET_CLDISCONNECTED; cdp.client = ci->client; cdp.timeout = true; SendAll((char*)&cdp, sizeof(ClDisconnectedPacket), true, false, &ci->addr); Client* c = &g_client[ci->client]; RichText msg = c->name + RichText(" timed out."); AddChat(&msg); } #else g_log<closed = true; //Close it using code below } //get rid of closed connections if(ci->closed) { if(&*ci == g_mmconn) { g_sentsvinfo = false; g_mmconn = NULL; } if(&*ci == g_svconn) g_svconn = NULL; #ifndef MATCHMAKER for(int cli=0; clion) continue; if(c->nc == &*ci) { if(g_netmode == NETM_HOST) { } if(c->player >= 0) { Player* py = &g_player[c->player]; py->on = false; py->client = -1; } c->player = -1; c->on = false; } } #endif //necessary to flush? already done in ReadDisconnectPacket(); //might be needed if connection can become ->closed another way. FlushPrev(&ci->addr); ci = g_conn.erase(ci); continue; } //inform other clients of unresponsive clients //or inform local player or unresponsive host if(now - ci->lastrecv > NETCONN_UNRESP && ci->isclient) //make sure this is not us or a matchmaker { #ifndef MATCHMAKER NetConn* nc = &*ci; Client* c = NULL; if(nc->client >= 0) c = &g_client[nc->client]; if(g_netmode == NETM_CLIENT && nc->isourhost) { //inform local player TODO c->unresp = true; } else if(g_netmode == NETM_HOST && nc->isclient && c) { //inform others if(c->unresp) { ci++; continue; //already informed } c->unresp = true; ClStatePacket csp; csp.header.type = PACKET_CLSTATE; csp.chtype = CLCH_UNRESP; csp.client = c - g_client; SendAll((char*)&csp, sizeof(ClStatePacket), true, false, &nc->addr); } #endif } ci++; } } Resending Packets Finally, ResendPacks(): void ResendPacks() { OldPacket* p; unsigned long long now = GetTicks(); //remove expired ack'd packets auto i=g_outgo.begin(); while(i!=g_outgo.end()) { p = &*i; if(!p->acked) { i++; continue; } //p->last and first might be in the future due to delayed sends, //which would cause an overflow for unsigned long long. unsigned long long safelast = enmin(p->last, now); unsigned long long passed = now - safelast; unsigned long long safefirst = enmin(p->first, now); if(passed < RESEND_EXPIRE) { i++; continue; } i = g_outgo.erase(i); } //resend due packets within sliding window i=g_outgo.begin(); while(i!=g_outgo.end()) { p = &*i; //kept just in case it needs to be recalled by other side if(p->acked) { i++; continue; } unsigned long long safelast = enmin(p->last, now); unsigned long long passed = now - safelast; unsigned long long safefirst = enmin(p->first, now); NetConn* nc = Match(&p->addr); //increasing resend delay for the same outgoing packet unsigned int nextdelay = RESEND_DELAY; unsigned long long firstpassed = now - safefirst; if(nc && firstpassed >= RESEND_DELAY) { unsigned long long sincelast = safelast - safefirst; //30, 60, 90, 120, 150, 180, 210, 240, 270 nextdelay = ((sincelast / RESEND_DELAY) + 1) * RESEND_DELAY; } if(passed < nextdelay) { i++; continue; } PacketHeader* ph = (PacketHeader*)p->buffer; /* If we don't have a connection to them and it's not a control packet, we need to connect to them to send reliably. Send it when we get a handshake back. */ if((!nc || !nc->handshook) && ph->type != PACKET_CONNECT && ph->type != PACKET_DISCONNECT && ph->type != PACKET_ACKNOWLEDGMENT && ph->type != PACKET_NOCONN) { Connect(&p->addr, false, false, false, false); i++; continue; } //do we want a sliding window? //edit: this is not correct, don't use this, it will cause a blockage #if 0 if(nc) { unsigned short lastack = nc->nextsendack + SLIDING_WIN - 1; if(PastAck(lastack, ph->ack) && ph->ack != lastack) { i++; continue; //don't resend more than SLIDING_WIN packets ahead } } #endif if(p->expires && now - safefirst > RESEND_EXPIRE) { i = g_outgo.erase(i); continue; } SendData(p->buffer, p->len, &p->addr, false, p->expires, nc, &g_sock, 0, NULL); p->last = now; i++; } } We 1.) erase OldPacket's that have been acknowledged (acked = true), 2.) check if the OldPacket in question is within the sliding window, and if it is, 2.) resend those OldPacket's that have reached a certain delay, 3.) and erase OldPacket's that are set to expire. "enmin" and "enmax" are just the min max macros: #define enmax(a,b) (((a)>(b))?(a):(b)) #define enmin(a,b) (((a)<(b))?(a):(b)) We don't want the "firstpassed" value (the amount of time that has passed since the OldPacket was first sent) to be negative (which would be a giant positive number for an unsigned 64-bit long long), so we set "safefirst" used in its calculation to be no more than the time "now", from which it is subtracted. If we didn't do this, we would get undefined behaviour, with some packets getting resent and some getting erased. unsigned long long safelast = enmin(p->last, now); unsigned long long passed = now - safelast; unsigned long long safefirst = enmin(p->first, now); NetConn* nc = Match(&p->addr); //increasing resend delay for the same outgoing packet unsigned int nextdelay = RESEND_DELAY; unsigned long long firstpassed = now - safefirst; Reading Acknowledgements Whenever we receive an AckPacket, we call ReadAckPacket on it in PacketSwitch: void ReadAckPacket(AckPacket* ap, NetConn* nc, IPaddress* from, UDPsocket* sock) { OldPacket* p; PacketHeader* header; for(auto i=g_outgo.begin(); i!=g_outgo.end(); i++) { p = &*i; header = (PacketHeader*)p->buffer; if(header->ack == ap->header.ack && Same(&p->addr, from)) { if(!nc) nc = Match(from); if(nc) { nc->ping = (float)(GetTicks() - i->first); } if(p->onackfunc) p->onackfunc(p, nc); i = g_outgo.erase(i); return; } } } In it, we will check for the matching buffered inbound OldPacket, and erase it from the list if found. But before that, we call a registered callback method that was set up when the packet was sent. Using the "first" time the packet was sent, subtracting it from the current time, gives the round-trip latency for that connection, which we can record in the NetConn class. Callbacks on Acknowledgement Whenever we send a DisconnectPacket, we set the callback function to: void OnAck_Disconnect(OldPacket* p, NetConn* nc) { if(!nc) return; nc->closed = true; //to be cleaned up this or next frame } Which will clean up the connection and stop resending the DisconnectPacket once it's acknowledged. It's best to encapsulate the needed functionality so we can safely Disconnect. void Disconnect(NetConn* nc) { nc->disconnecting = true; //check if we already called Disconnect on this connection //and have an outgoing DisconnectPacket bool out = false; for(auto pit=g_outgo.begin(); pit!=g_outgo.end(); pit++) { if(!Same(&pit->addr, &nc->addr)) continue; PacketHeader* ph = (PacketHeader*)pit->buffer; if(ph->type != PACKET_DISCONNECT) continue; out = true; break; } if(!out) { DisconnectPacket dp; dp.header.type = PACKET_DISCONNECT; SendData((char*)&dp, sizeof(DisconnectPacket), &nc->addr, true, false, nc, &g_sock, 0, OnAck_Disconnect); } } When we receive an acknowledgement of a ConnectPacket that we sent out, we also need to set "handshook" to true. You can set user callbacks for certain special connections, like matchmakers or game hosts, to carry out certain functions, like immediately polling for servers, or getting server info, or joining the game room. //on connect packed ack'd void OnAck_Connect(OldPacket* p, NetConn* nc) { if(!nc) nc = Match(&p->addr); if(!nc) return; nc->handshook = true; ConnectPacket* scp = (ConnectPacket*)p->buffer; //if(!scp->reconnect) { #ifndef MATCHMAKER GUI* gui = &g_gui; if(nc->isourhost) { g_svconn = nc; //TO DO request data, get ping, whatever, server info JoinPacket jp; jp.header.type = PACKET_JOIN; std::string name = g_name.rawstr(); if(name.length() >= PYNAME_LEN) name[PYNAME_LEN] = 0; strcpy(jp.name, name.c_str()); jp.version = VERSION; SendData((char*)&jp, sizeof(JoinPacket), &nc->addr, true, false, nc, &g_sock, 0, NULL); } #endif if(nc->ishostinfo) { //TO DO request data, get ping, whatever, server info GetSvInfoPacket gsip; gsip.header.type = PACKET_GETSVINFO; SendData((char*)&gsip, sizeof(GetSvInfoPacket), &nc->addr, true, false, nc, &g_sock, 0, NULL); } #ifndef MATCHMAKER if(nc->ismatch) { g_mmconn = nc; g_sentsvinfo = false; if(g_reqsvlist && !g_reqdnexthost) { g_reqdnexthost = true; GetSvListPacket gslp; gslp.header.type = PACKET_GETSVLIST; SendData((char*)&gslp, sizeof(GetSvListPacket), &nc->addr, true, false, nc, &g_sock, 0, NULL); } } #endif } } You can see there's a commented out function pointer called "chcallback" in the NetConn class, which might be given a function to call when the connection is handshook, instead of hard-coding several cases for the connection type ("ismatch", "isourhost", etc.) Connecting Before we host a server or connect to the matchmaker, we must open a socket. void OpenSock() { unsigned short startport = PORT; if(g_sock) { IPaddress* ip = SDLNet_UDP_GetPeerAddress(g_sock, -1); if(!ip) g_log<<"SDLNet_UDP_GetPeerAddress: "<port); SDLNet_UDP_Close(g_sock); g_sock = NULL; } if(g_sock = SDLNet_UDP_Open(startport)) return; //try 10 ports #ifndef MATCHMAKER for(int i=0; i<10; i++) { if(!(g_sock = SDLNet_UDP_Open(PORT+i))) continue; return; } #endif char msg[1280]; sprintf(msg, "SDLNet_UDP_Open: %s\n", SDLNet_GetError()); g_log< This OpenSock method will try 10 different port numbers if the first one fails. If it still doesn't work, it will log a message from SDLNet. After we open a port, we can send packets. The OpenSock method is encapsulated in the Connect() function. We can call the first Connect method, which takes an IP string or domain address, or the second, which accepts an IPaddress struct. They also accept some parameters to describe their use, like whether the connection is the matchmaker, the host being joined, a client of our room, or a random server we're getting info on. NetConn* Connect(const char* addrstr, unsigned short port, bool ismatch, bool isourhost, bool isclient, bool ishostinfo) { IPaddress ip; //translate the web address string to an IP and port number if(SDLNet_ResolveHost(&ip, addrstr, port) == -1) { return NULL; } //call the following function... return Connect(&ip, ismatch, isourhost, isclient, ishostinfo); } //Safe to call more than once, if connection already established, this will just //update NetConn booleans. NetConn* Connect(IPaddress* ip, bool ismatch, bool isourhost, bool isclient, bool ishostinfo) { if(!g_sock) OpenSock(); NetConn* nc = Match(ip); NetConn newnc; bool isnew = false; //if we don't recognize this address as having a connection to, make a new NetConn instance for the list if(!nc) { isnew = true; newnc.addr = *ip; newnc.handshook = false; newnc.lastrecv = GetTicks(); newnc.lastsent = newnc.lastrecv; //important - reply ConnectPacket with ack=0 will be //ignored as copy (even though it is original) if new NetConn's lastrecvack=0. newnc.lastrecvack = USHRT_MAX; newnc.nextsendack = 0; newnc.closed = false; g_conn.push_back(newnc); nc = &*g_conn.rbegin(); } else { //force reconnect (sending ConnectPacket). //also important for Click_SL_Join to know that we //can't send a JoinPacket immediately after this function, //but must wait for a reply ConnectPacket. if(nc->closed) nc->handshook = false; } bool disconnecting = false; //if we have an outgoing DisconnectPacket, set disconnecting=true for(auto pit=g_outgo.begin(); pit!=g_outgo.end(); pit++) { OldPacket* op = &*pit; if(!Same(&op->addr, &nc->addr)) continue; PacketHeader* ph = (PacketHeader*)op->buffer; if(ph->type != PACKET_DISCONNECT) continue; disconnecting = true; break; } //if we're closing this connection, don't send any other reliable packets on it except DisconnectPacket and clear any outbound or inbound OldPacket's if(disconnecting) { nc->handshook = false; FlushPrev(&nc->addr); } //different connection purposes //only "true" it, or retain current state of nc->... nc->isclient = isclient ? true : nc->isclient; nc->isourhost = isourhost ? true : nc->isourhost; nc->ismatch = ismatch ? true : nc->ismatch; nc->ishostinfo = ishostinfo ? true : nc->ishostinfo; if(isourhost) g_svconn = nc; if(ismatch) g_mmconn = nc; //see if we need to connect for realsies (send a ConnectPacket). //i.e., send a connect packet and clean previous packets (OldPacket's list). if(!nc->handshook) { bool sending = false; //sending ConnectPacket? unsigned short yourlastrecvack = PrevAck(nc->nextsendack); //check if we have an outgoing ConnectPacket for(auto pi=g_outgo.begin(); pi!=g_outgo.end(); pi++) { if(!Same(&pi->addr, &nc->addr)) continue; PacketHeader* ph = (PacketHeader*)pi->buffer; if(PastAck(PrevAck(ph->ack), yourlastrecvack)) yourlastrecvack = PrevAck(ph->ack); if(ph->type != PACKET_CONNECT) continue; sending = true; break; } if(!sending) { ConnectPacket cp; cp.header.type = PACKET_CONNECT; cp.reconnect = false; cp.yourlastrecvack = yourlastrecvack; cp.yournextrecvack = nc->nextsendack; cp.yourlastsendack = nc->lastrecvack; SendData((char*)&cp, sizeof(ConnectPacket), ip, isnew, false, nc, &g_sock, 0, OnAck_Connect); } } nc->closed = false; return nc; } When closing a connection, or connecting again after a connection had been disconnected, we flush any buffered in- or out-bound OldPacket's. //flush all previous incoming and outgoing packets from this addr void FlushPrev(IPaddress* from) { auto it = g_outgo.begin(); while(it!=g_outgo.end()) { if(!Same(&it->addr, from)) { it++; continue; } it = g_outgo.erase(it); } it = g_recv.begin(); while(it!=g_recv.end()) { if(!Same(&it->addr, from)) { it++; continue; } it = g_recv.erase(it); } } We read a ConnectPacket like this: void ReadConnectPacket(ConnectPacket* cp, NetConn* nc, IPaddress* from, UDPsocket* sock) { bool isnew = false; if(!nc) { nc = Match(from, cp->header.senddock); if(!nc) { nc = Match(from, 0); if(nc) nc->dock = cp->header.senddock; } if(!nc) { isnew = true; NetConn newnc; newnc.addr = *from; newnc.handshook = true; newnc.lastrecvack = cp->header.ack; newnc.nextsendack = 0; newnc.lastrecv = GetTicks(); newnc.closed = false; g_conn.push_back(newnc); nc = &*g_conn.rbegin(); } } if( nc && ( nc->ismatch || nc == g_mmconn ) ) { nc->ismatch = true; g_mmconn = nc; g_sentsvinfo = false; } nc->handshook = true; nc->closed = false; if(isnew) { nc->lastrecvack = cp->header.ack; nc->nextsendack = 0; } else { FlushPrev(&nc->addr); nc->lastrecvack = cp->header.ack; nc->nextsendack = 0; } #endif } And the DisconnectPacket: void ReadDisconnectPacket(DisconnectPacket* dp, NetConn* nc, IPaddress* from, UDPsocket* sock) { if(!nc) nc = Match(from, dp->header.senddock); if(!nc) return; if(nc->isourhost) { g_svconn = NULL; #ifndef MATCHMAKER EndSess(); RichText mess = STRTABLE[STR_HOSTDISC]; //host disconnected Mess(&mess); #endif //TODO message box to inform that host left the game and that game is over } if(nc->ismatch) { g_mmconn = NULL; g_sentsvinfo = false; } for(auto ci=g_conn.begin(); ci!=g_conn.end(); ci++) if(&*ci == nc) { ci->closed = true; FlushPrev(&ci->addr, ci->dock); break; } #ifndef MATCHMAKER //get rid of client if(nc->client >= 0) { Client* c = &g_client[nc->client]; if(g_netmode == NETM_HOST) { //inform other clients ClientLeftPacket clp; clp.header.type = PACKET_CLIENTLEFT; clp.client = nc->client; SendAll((char*)&clp, sizeof(ClientLeftPacket), true, false, &nc->addr, nc->dock); RichText msg = c->name + STRTABLE[STR_LEFT]; AddChat(&msg); } ResetCl(c); } #endif nc->client = -1; } Conclusion That is all for this article. If you want to see the rest of the article covering parity bit checking, lockstep, and LAN networking, purchase the full article here: http://www.gamedev.net/files/file/223-reliable-udp-implementation-lockstep-lan-and-parity-bit-checking/ Article Update Log 4 Apr 2016: Forgot to tell you how to respond to a ConnectPacket and DisconnectPacket. 29 Dec 2015: Fixed some bugs. The first line here // Translate in order if(checkprev && nc) { while(true) { if(!Recieved(last+1, last+1, nc)) break; last++; ParseRecieved(last, last, nc); } } Should be if(checkprev && nc && last == header->ack) Also, commented out the sliding window because it is an incorrect implementation and causes a blockage. And in Recieved() the line if(missed) should be if(missed && current != afterlast) I changed how OldPacket's are added to g_recv and g_outgo in AddRecieved and SendData. And this header = (PacketHeader*)&p->buffer; Is now this header = (PacketHeader*)p->buffer; 6 Oct 2015: Initial release