# Proper sendto() Error Handling

This topic is 2767 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

What are the best/most common/standard ways to handle errors returned by sendto()?

In WinSocks, the documentation says it can return a length that is smaller than the input length.
http://msdn.microsoft.com/en-us/library/ms740148(v=vs.85).aspx
So? Should I ignore that? Should I try to sendto() the remaining packets?

If I detect WSAEWOULDBLOCK and WSAEINPROGRESS, I wait 2 milliseconds and try again.
Is this standard?
All other error codes are treated as errors, and WSAECONNRESET and WSAECONNABORTED indicate that the socket should be closed.
How about it? Anything wrong? Anything more I need?

Yogurt Emperor

##### Share on other sites

What are the best/most common/standard ways to handle errors returned by sendto()?

Generally, you want the networking code to be double-buffering with your game code.

The game code will do two things each frame/loop/tick:

1) Look to see whether there's enough incoming data in the in buffer for a full packet, and if so, decode and handle that packet. Repeat.
2) Put a copy of any packet destined to go out in the outgoing queue.

The networking code will then do two things:

1) recv() as much data as it can in a single sweep into the incoming buffer. You may wish to limit the amount of data recv()-ed per frame/loop to something like 4k or 8k. This data will be visible to the game code during the next frame.
2) send() as much data as it can in a single sweep out of the outgoing buffer. The implementation may or may not accept all of the data you try to push at it, but that's OK, because the rest is still in the buffer.

If the incoming or outgoing buffers become too full (say, bigger than 32 k, or some other arbitrary limit), then the other end is mis-behaving, and your best bet is to just disconnect it before it consumes too much memory.

##### Share on other sites
So a sendto() that returns less than the input length means it sent some data but not all, and I need to find a way to send the rest (which will be through double-buffering as you say).
But what implications are there for UDP connections which can send data out-of-order?

This means the second half of the data may arrive before the first, and I have to also implement a way to manage this situation, correct?

Yogurt Emperor

##### Share on other sites
In general, you always want to be aware of out of order packets. A simple sequence number included in each packet allows the remote side to drop out of order packets.

For UDP, you want a fixed upper limit to your packets. You do this anyway to limit fragmentation at the lower network layers. If you are trying sendto() and not everything is being sent, then this will cause problems unless you have built a reliable packet layer on top.

It is generally inadvisable for a broadcast datagram to exceed the size at which fragmentation can occur, which implies that the data portion of the datagram (excluding headers) should not exceed 512 bytes.
[/quote]

##### Share on other sites
Yes, I am planning all that.

I have just never done networking coding before and want to understand the API and all the little details that go into the big implementation first. Very important for a solid foundation, and a solid foundation is very important for my task (a next-gen commercial game engine).

I have the full big picture after reading many articles on the subject.
As I get into the implementation, I encounter these small details, such as what the actual meaning of a sendto() return being less than the buffer size, what to do about it, how often it can happen, etc.

Although the article states that being over 512 bytes can cause fragmentation, I don’t want to assume that fragmentation can not occur below that.
How often, if ever, does fragmentation occur below 512 bytes of input?

If it truly is never, I don’t need to design a fragmentation handler, but I will be stuck with packets under 512 bytes.

I suppose that handling fragmentation is still the best choice in the long run.

http://trac.bookofhook.com/bookofhook/trac.cgi/wiki/IntroductionToMultiplayerGameProgramming
He says it is tricky to get working while avoiding race conditions, deadlocks, etc.
These things are non-issues at my level. Should I still avoid threading? To me I think it would simplify a few things and make some things smoother, but again my knowledge of networking is naive, and I worry about getting bitten in the ass by some unforeseen hiccup.
Anyone have experience putting the network code on another thread?

Yogurt Emperor

##### Share on other sites

I have just never done networking coding before and want to understand the API and all the little details that go into the big implementation first. Very important for a solid foundation, and a solid foundation is very important for my task (a next-gen commercial game engine).

If this is the case, you will definitely want to look at existing UDP libraries used by games to get an idea of what they do and what features they provide. Namely, RakNet (complete game library), enet (low level wrapper), lidgren, NetDog (caters to MMOs), SmartFoxServer (a whole platform), Photon (another platform), boost::asio (cross platform wrapper library), POCO (similar to boost libraries), and finally ACE (toolkit).

Focus on the big picture first. The "little details" come about from the specific network implementation you use. I.e., using regular Winsock, POSIX, BSD sockets come with their own little quirps and platform specific limitations, so if your overall goal is a next-gen commercial game engine, then the platforms you will support will play a large part into what you might want to build upon.

For example, if you have no plans for console support, then there is no reason not to start looking over boost or poco which contain a lot of code dealing with the little things already. The research and development has been done, so you don't need to discover it all yourself as much as just look at existing solutions to see what is done. That is not to say you can't code your own implementation, you can if you so choose, but trying to relearn decades of R&D from scratch is not worth it.

If you do plan on supporting consoles, then things like boost or poco are no good to you, so you already know there's no real need for them because you don't want different networking cores for different platforms (imo). You can still consult them for PC specific stuff, but you'd probably want to go for something a lot more lightweight depending on all the languages you will have to support for your engine.

http://trac.bookofho...GameProgramming
He says it is tricky to get working while avoiding race conditions, deadlocks, etc.
These things are non-issues at my level. Should I still avoid threading? To me I think it would simplify a few things and make some things smoother, but again my knowledge of networking is naive, and I worry about getting bitten in the ass by some unforeseen hiccup.[/quote]

That unforeseen hiccup is generally " race conditions, deadlocks, etc.". "Stuff" happens, even when you are familiar with the concepts.

All it comes down to is "shared state" really.

This is more of a bigger issue with TCP than it is with UDP. Since TCP is a stream protocol, if you have multiple recvs posted on a socket, you must maintain the order that the data is returned before processing it. So if thread 1 recvs data into a buffer object with a sequence id of N and thread 2 recvs data into a buffer object with sequence id N + 1, then you must be sure to process the object with id N before N + 1. This means you have to code a custom system to handle this if you need that type of functionality. Unless you are working with massive data transfers using hardware that is meant for it, then usually it's just easier to keep one recv posted on a socket at a time and only process data from one thread. That one thread might be a worker thread in a pool of many, so it may not be the same each time, but only one concurrent recv is posted at a time.

Other designs dispatch the packet processing logic from the network thread. This design only works for systems with very little shared state (as to not tie up the network thread) and very lightweight logic processing. For example, a simple web server with no shared sate between the connections might take this approach. Since the context object will not be accessed from any other part of the system, it works out 'ok'. If you were trying to write a MMO, then this design is not practical because you tie up a network worker thread (which if your thread pool does not grow, can be a real problem). It's far easier to deadlock the threads with a lot of shared state locks so by keeping them to a minimal (i.e. packet queue lock then dispatch) you reduce the problems you might have.

With UDP, you generally only need one main thread to handle the server socket. This is because you are not working with "connections" like you are in TCP. As a result, when you get a packet in the network thread, you can simply pass it to another thread to take care of processing it. In other words, the network thread is only responsible for pulling packets from the wire and passing them along. For this to work out, you have to make sure the method you are using to pass the packets along does not involve a lot of overhead.

For example, let's say you recvfrom one packet. You lock the queue, add the packet, and continue on. If the thread that is processing the packets acquires the lock but does not release it fast enough by the time the next packet is processed, then you stall the network thread some each time and really don't gain anything from this setup. That is why, going back to the TCP example, when you lock the queue, copy it, and release, you minimize the contention and can maintain a well functioning system in that regards.

Depending on the platform and language, you might have to optimize the "copy" step to make sure you don't get hit with unneeded global locks for allocation or deep copying. Even so, that's an optimization that only should be pursued after profiling and having solid evidence it's causing issues! Likewise, the lock mechanisms you use do matter. If you are on a platform where the most viable locks incur significant overhead, then the benefits of using multiple threads is greatly reduced.

Once you managed to get your packets out of the network layer, then you are back to dealing with your typical issues of message validation, multi-threaded programming, and so on.

Working out how you "send" data through the network might be more troublesome with some designs then others. For example, you never want to allow threads to arbitrarily send data instantly. If you do, you just make life significantly harder than you have to when things go wrong. Instead, you want to save outgoing messages to a queue and process them from a specific context. The implications of doing this though can be quite complex.

For example, with TCP, you have to wait until all the previous buffered data is passed through send before starting the next send. As a result, you must make sure you chain your send triggering logic so you don't end up with data still in a send buffer waiting to be passed through send, but no event is pending to trigger it. This is more related to the multi-threaded programming issues than just networking. With UDP, you just have to make sure that up until the packet data is dispatched with sendto,you maintain protocol synchronization so you don't have a case where you send a packet with sequence number N, but a security flag based on sequence number N + 1 as a result of not handling your locks the right way (i.e., locking individual pieces of logic instead of the entire unit). If you don't have any extra protocol specific security stuff, then it's not as big of an issue, but any decent system should.

Of course, if you are only using one thread and very simple networking APIs, like select, you don't have all the headaches to deal with when it comes to multi-threaded programming, so keep that in mind. I'd say start simple and small and see how it carries you. I've found it's nice to have solutions for both single and multi-threaded setups, so don't feel like you have to just dive into making an uber multi-threaded solution whose power and functionality might not ever be utilized due to other constraints. In the end, a lot of what you do is going to depend on your target platforms and the language you use.

Lastly, since I couldn't fit it in anywhere else, you have to keep in mind the data type differences and byte order of different platforms and architectures. You must code your higher level network logic to take this into account. Otherwise, you will run into tons of (not so) fun issues that are really hard to figure out after the fact. So keep in mind: size_t size differences, wchar_t size differences, float/double precision differences, code page string processing differences, and many more. A lot of the things you have to be worried about are not directly related to network API issues in network programming!

##### Share on other sites
That’s a mouthful.

Thank you for the long post and it was worth the read.
My engine supports PC (OpenGL, Direct3D 9, Direct3D 10, and Direct3D 11 also, as well as x86 and x64), Linux, Macintosh, iOS, Nintendo Wii, Xbox 360, and PlayStation 3. And is planned to be easily upgradable for the next generation of consoles, whatever features they may have.
Imagine CryEngine 3 running on every platform, available to the public, and at a much lower price (planning on $250 for indies). That is the goal here. So I am accustomed to data being in different endians or sizes. I already have types and macros defined for it, so that’s out of the way at least. I am confident in my ability to avoid deadlocks and race conditions, but worried about performance. Based off your advice, I will start by coding a single-threaded networking system that can be completely #ifdef’d out. By not defining LSE_SINGLE_THREADED_NETWORKING, I will have NO networking functionality. Then I can build a second multi-threaded system, and easily switch between them. No game will need both systems at once. I will check out the references you suggested. Yogurt Emperor #### Share this post ##### Link to post ##### Share on other sites Imagine CryEngine 3 running on every platform, available to the public, and at a much lower price (planning on$250 for indies). That is the goal here.

That's a fine goal! Will it also have the Sandbox style editor?

[color=#1C2837][size=2]But what implications are there for UDP connections
[color=#1C2837][size=2][/quote]

For a UDP socket, a single call to send() will either send all of the data as a single datagram, or send none of the data. There are no partial datagrams in UDP. And "send" really means "put in the kernel output buffer."

If you're targeting Xbox and PS/3, then you're probably aware of the additional network API requirements for certification on those platforms, but I thought I should mention them in this context, because if you want a single API across all, you need to understand the requirements and semantics of all the APIs in detail in order to not paint yourself into a corner.

##### Share on other sites

[quote name='YogurtEmperor' timestamp='1305297970' post='4810236']
Imagine CryEngine 3 running on every platform, available to the public, and at a much lower price (planning on \$250 for indies). That is the goal here.

That's a fine goal! Will it also have the Sandbox style editor?

[color="#1C2837"]But what implications are there for UDP connections
[color="#1C2837"] [/quote]

For a UDP socket, a single call to send() will either send all of the data as a single datagram, or send none of the data. There are no partial datagrams in UDP. And "send" really means "put in the kernel output buffer."

If you're targeting Xbox and PS/3, then you're probably aware of the additional network API requirements for certification on those platforms, but I thought I should mention them in this context, because if you want a single API across all, you need to understand the requirements and semantics of all the APIs in detail in order to not paint yourself into a corner.
[/quote]

I am planning the Sandbox editor, but possibly not in the first release.
Initially, I plan to rely on [font=arial, sans-serif][size=2]Valve Hammer Editor and existing terrain editors, but only by converting their formats to my own. Then I can slide them out of the picture smoothly and make my own editors later which export to my format directly, and work more like Sandbox.[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]As for additional network requirements for certain platforms, I am not aware.[/font]
[font=arial, sans-serif][size=2]Network coding is the last part of engine coding that I have never done, at all.[/font]
[font=arial, sans-serif][size=2]I plan to mitigate surprises such as these by first making wrappers that create a single common interface for each network API I use and keeping it as minimalistic as possible.[/font]
[font=arial, sans-serif][size=2]As long as each new API can somehow replicate the minimal interface I have created, I won’t have too many problems. The rest of the network library will use only my custom interface.[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]That is the plan anyway. Feel free to point out anything about consoles that may give me headaches if I don’t prepare for them now.[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]I am only using UDP.[/font]
[font=arial, sans-serif][size=2]So I never have to worry about sendto() returning a number less than the input length (fragmentation)?[/font]
[font=arial, sans-serif][size=2]Even if I sendto() 2,048 bytes? It is guaranteed to arrive as a 2,048-byte chunk on the other end?[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]
[/font]
[font=arial, sans-serif][size=2]Yogurt Emperor[/font]

##### Share on other sites

[font="arial, sans-serif"]So I never have to worry about sendto() returning a number less than the input length (fragmentation)?[/font]
[font="arial, sans-serif"]Even if I sendto() 2,048 bytes? It is guaranteed to arrive as a 2,048-byte chunk on the other end?[/font]

The maximum UDP datagram payload length is 65,507 bytes (given the underlying IP datagram size limitations). If IP ends up fragmenting the IP datagram, then that is transparent to the UDP layer, insofar as the entire packet is accepted for delivery, and the entire packet is delivered, or nothing at all. However, many routers or firewalls deal poorly with fragmented IP packets, so there is a "suggested" maximum limit on the order of <1,500 bytes of payload. Note that some UDP protocols, such as NFS, specify packets somewhat larger than 8kbytes, thus split into six Ethernet frames, so IP fragmentation is certainly in use in some large systems in the world today.

If you sendto() for 2048 bytes, then 0 or 2048 bytes will be accepted and delivered. Read up on the blocking semantics of sendto() and how it interacts with select() and potentially making sockets non-blocking for details. On the other end, those 2048 bytes may be delivered all-or-nothing zero or more times, perhaps out of order. If IP fragment reassembly fails, it means zero times. If the internet has a hiccup, it may mean 2 or 3 times, perhaps later than some other packets that were sent in another order.

1. 1
2. 2
Rutin
29
3. 3
4. 4
5. 5
khawk
14

• 11
• 11
• 23
• 10
• 9
• ### Forum Statistics

• Total Topics
633648
• Total Posts
3013111
×