Multithreaded server - horrible performance

Started by
27 comments, last by LycaonX 14 years ago
Quote:Original post by LycaonX
... I synclock m_Packets.GetType, add the client/data, then end synclock...


But if it's just a typo then never mind :-)
Advertisement
Are you running the server on a home connection? Around 25 players might be where you run out of up bandwidth. Not sure how that would lag a local connection though.
My c#.Net server uses the Begin* methods, with very little lag (Though I havn't been able to really stress test it as much as I would like).
Are you on .Net 3.5? When I upgraded from 2.0 to 3.5 there was a slight increase in processing time for sockets. Also, 3.5 introduces a newer, more C like implementation (which I have not taken advantage of yet).

Look Here:
http://msdn.microsoft.com/en-us/magazine/cc163356.aspx
Quote:Original post by hplus0603
It's hard to tell how you arrived at those numbers


Nothing fancy, just grabbing Environment.TickCount immediately before and after each packet is processed.

As far as connection, I'm on what most would consider a fairly low capacity upload (896 kbps) but the most my server's ever sent out over the DSL is ~25 kilobytes per sec. 90% of the traffic is from the NPCs which are either running on the same computer as the server, or on a LAN computer.

I messed around with it a bit last night and I was (randomly?) getting anywhere from 10 to 300 ms ping from the client on my laptop to the server on my desktop.

It's running on top of 3.5, I'll see if 2.0 makes any difference.
If you're on Vista or 7, you should use the Performance Monitor to monitor incoming and outgoing network traffic.

Also, TickCount has terrible resolution -- it may update only once every 20 milliseconds or so. For timing, use a System.Diagnostics.Stopwatch.

In general, a server using BeginReceve()/EndReceive() should have no problems processing thousands of simultaneous connections, at least when running on Windows Server. There may be artificial limitations on the consumer levels of Windows.
enum Bool { True, False, FileNotFound };
I switched to the .XXXAsync methods with a pool of SocketAsyncEventArgs. According to the link posted above, those methods are specifically designed for high traffic, high performance server applications in .NET. I think I've seen a definite decrease in 'lag' but there's still enough to alarm me.

I seem to recall reading somewhere that .NET sockets have a small delay between queueing data to be sent and actually sending it (if the data is extremely small, like my 8 byte 'ping' packets). This might also be a serious contributor to the latency, though I doubt the entire 1-3 seconds is solely the fault of this. Do any of you know what I'm talking about? Or is it some jumbled mix of different articles I read?

Also, in reply to the above post, I'm on XP 64 Pro but I do believe I've used permon on it before. I'll see what I can dig up with it.

EDIT: Found this on MSDN: To increase network efficiency, the underlying system may delay transmission until a significant amount of outgoing data is collected. A successful completion of the BeginSend method means that the underlying system has had room to buffer your data for a network send. If it is important to your application to send every byte to the remote host immediately, you can use SetSocketOption to enable SocketOptionName.NoDelay.

I am going to go ahead and try this. It's fully possible that the 8 byte Ping packets are sitting there waiting for a bigger chunk of data to push them out to the client. If this is so, I've been freaking out over nothing major. :)
The difference between Begin...() and ...Async() is not going to matter in your case, because you are not CPU bound.

If you haven't yet turned on NoDelay, then you should -- that's the same as TCP_NODELAY mentioned in the Forum FAQ. However, that would affect the very first client -- not after 25 have connected.

I still think you are saturating your network somewhere, or running into some bug in the implementation of the server, if there is a point where lag starts increasing. You can use something like Wireshark to look for TCP retransmits to see whether you start dropping packets.
enum Bool { True, False, FileNotFound };
I can't imagine I'm saturating either a gigabit connection (when testing with the npcs on one computer and the server on this one) nor local loopback (running both on the desktop); I even track incoming and outgoing data for both LAN and internet clients and about the highest I've seen is 30kbps (LAN clients) and ~65kbps (internet clients). If the clients are not in the same room (this is the majority of cases), bandwidth use is negligible at about 150 bytes per second per client. This increases exponentially for each additional client in the same room, topping out my DSL upload at about 25 people in the same room (rare).

But yeah, looking more and more like it might be an implementation problem. I did set .NoDelay and still have the problem. I guess I'll dig out Wireshark and peek at traffic. Wish I knew how to filter out packets based on a certain value in the data :p

[Edited by - LycaonX on April 3, 2010 8:34:07 PM]
So I hate to admit this, but I don't want to leave a thread hanging.

I ran Wireshark and looked at traffic and although I was not getting anywhere close to saturating my connection, I *DID* notice that the server was sending out massive quantities of pings. I looked at my code and sure enough, the ping code was sitting -outside- of an if block that checked for elapsed time. Due to this, my server was sending out pings to all npc clients once per second, and as they started to get backlogged, the problem was compounded because the server noticed that it hadn't received a ping reply.

All in all, I'm glad I ran into this problem, as I did end up learning some more stuff about the framework in general, as well as some extra goodies in the last version :) The .XXXAsync methods show a consistant 20% decrease in processing time when handling over 400 clients, so this wasn't a total facepalm.

Thanks to everyone that commented and helped out, I hope I can return the favor to the forums some day :)

This topic is closed to new replies.

Advertisement