So, after weeks of strange and seemingly intermittent/coincidental UDP server crashes(Receive Thread Halt/Un-catchable Exception) and the ensuing frustration that that causes when you aren't even trying to work on the server itself.  I sat down and spent a day trying to figure out the problem..  After an almost complete server rewrite from scratch with no real improvement in the problem.. I turned to random poking about on the internets..

Hmmm...  It seems like the problem only happens when I'm restarting the client..

Here's a post!!

https://stackoverflow.com/questions/38191968/c-sharp-udp-an-existing-connection-was-forcibly-closed-by-the-remote-host

And Here!

https://social.msdn.microsoft.com/Forums/en-US/6b7aa353-259b-4637-b0ae-d4e550c13e38/c-udp-socket-exception-on-bad-disconnect?forum=netfxnetcom

Yup, that's the problem for sure.

It appears that even though UDP isn't supposed to care about the other ends state, apparently this isn't true if the state is Port Unreachable.  When the server attempts to receive data (BeginReceiveFrom/BeginReceive/EndRecieveFrom/EndReceive/etc..) from an async connection and the client has (Crashed/Quit/Etc.) an ICMP port unavailable packet is generated and this causes the UDP socket to generate an exception that (even if you try/catch the hell out of it) causes the async receive thread to halt(I assume, this is the effect anyhow).

The best solution I've found, as stated in the link(s) above is to change the underlying IOControl behavior(Prevent the Exception from being thrown).

//This value tells the underlying IO system to IGNORE the ICMP error.
public const int SIO_UDP_CONNRESET = -1744830452;
private Socket _socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);

_socket.IOControl((IOControlCode)SIO_UDP_CONNRESET, new byte[] { 0, 0, 0, 0 }, null);
_socket.Bind(new IPEndPoint(IPAddress.Any, port));

The first line is the value that needs to be set, and the second to last line is how you can set it.  And these are the live bits of code that are WORKING without error in my server, Finally! 

So, there, I've posted it again because it seems way more obscure than it should be.

Enjoy.