My game server is eating my CPU

Started by
26 comments, last by rip-off 12 years, 8 months ago
Hi

I made a multiplayer 3d game and the server eats up way too much CPU. On a quad-core 3Ghz machine it only supports 5 players before things get realllll slooooooowwww.

The server is multithreaded, one thread per player and checks the line for input 60 times a second.

Does this make sense? Is my design somehow flawed? Does anybody know how I can up the number of players?

I can't check the line less often coz then player movement is disrupted.

Thanks
Advertisement
one thread per player
Networking and Multiplayer FAQ -- Question 10 (Should I spawn a thread per connection in my game code?).
Having one thread per player is likely slower, more complex and more bug-prone than just using a single thread for all players.
You can read data from your sockets in a non-blocking manner, so that the thread doesn't stall when there's no data from a particular player.
checks the line for input 60 times a second.[/quote]Many action games, like Counter-Strike, only update 20 times per second, and interpolate between results. Perhaps you can describe your algorithms/architecture a bit?
I'll take a look at the FAQ, thanks. But my sockets are already non-blocking.

The server checks for update every 1/60 second, when it gets an instruction (key_was_pressed) it calculates object's new position and sends an update to all players.

Hi

I made a multiplayer 3d game and the server eats up way too much CPU. On a quad-core 3Ghz machine it only supports 5 players before things get realllll slooooooowwww.

The server is multithreaded, one thread per player and checks the line for input 60 times a second.

Does this make sense? Is my design somehow flawed? Does anybody know how I can up the number of players?

I can't check the line less often coz then player movement is disrupted.

Thanks


Figure out WHY things get slow, then fix those things.

You say it "eats up way too much CPU". That probably means the problem is not the networking. Throwing a few buffers to your network card is not a CPU hog.

Use a profiler to figure out what is consuming all the time in your program. The "Premium" and "Ultimate" levels of Visual Studio include a profiler. Visual Studio also has a general CLR profiler that is moderately okay. You can get free profilers such as CodeAnalyst for C++ or if you are using a managed/.NET environment, tools like nprof, slimtune, and the eqatac profiler are available.

Look over your profile results carefully. Be sure you are measuring the optimized version of your code, and that you measure both before AND after making changes.



Using a profiler is pretty easy, there are many tutorials out there. Basically you build your program with some special libraries and run it. At some point you use a tool (such as your debugger) to start profiling, let it run for a few seconds, then stop profiling. The system figures out the results and gives you a bunch of tables showing the functions that were called, how long they took, and how many times they were called.

Profiling networked code is not so different than local code other than it takes a while to set up the normal work conditions of connecting multiple machines and getting to the problem spots.

If you haven't profiled your code before, usually the first time through you notice obviously stupid things like calling "strlen()" a billion times each update, or running an A* pathfinding on hundreds of items every frame, or similar accidental mistakes that you just didn't realize were there. From there you backtrack to why you are calling strlen a billion times and change the code to only call it a few times. Or you look at the code that is running pathfinding on every object every frame, and prune it down to running it only on a few items at a time. Or you'll notice that moving an object runs a query on every other game object instead of just those nearby. Or you'll notice other entirely different things.

Profiling and improving code is a learned art. You need to discover what is slow, and replace it with something faster. Generally, fixing profiler-discovered hot spots means reusing or caching values, spreading work across time, or changing your algorithm to one that processes less data. Sometimes the improvements are as simple as moving code outside a loop. Other times they require significant work writing tools to preprocess data so it can be managed more quickly in game, such as converting a basic mesh map into a spatial tree. Every issue is different, and has its own unique challenge.
Your networking threads should be using relatively low amounts of CPU. If they aren't, something is wrong. If you are using a non-blocking socket in a thread, they are probably spinning on the socket as fast as possible looking for incoming messages, which is needlessly burning CPU time.

As mentioned, you generally don't want a "thread per client", but perhaps switching to blocking reads might allow you some breathing space until you have time to make the larger architectural changes.
Do you have a "Sleep(1)" at the end the server update function? That should make it much nicer to the CPU.

You running this on a home network? Consumer broadband in most places has really bad up speeds, which is what a game server needs the most. You mention "5 players and it starts crapping out" so that might be where you are hitting your up rate limit.

Also, do you really need 60 updates a second? Try dropping it a bit.

Do you have a "Sleep(1)" at the end the server update function? That should make it much nicer to the CPU.


Sleeping is the wrong way to relieve CPU usage, especially in a network server.

The correct approach is to directly wait on the sockets, either using blocking modes, IOCPs, or select() with a timeout, or whatever. Just sleeping and then busy-polling the socket is incredibly bad design.

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]

Thanks you all for your suggestions.

I'm doing a Sleep(33) on every thread to regulate to 30 recv()s per second, apparently non-blocking ones. What is IOCP? I'm running locally using the loopback interface. Now I can support 10-15 users, and the limiting factor is probably client use of CPU (unlike before when the server was the hog). I'll have to find another machine to do a more realistic test. But it's already much better :) thanks
[font="arial, verdana, tahoma, sans-serif"]

If you are using a non-blocking socket in a thread...

[/font]Do you have a "Sleep(1)" at the end the server update function? That should make it much nicer to the CPU.

You running this on a home network? Consumer broadband in most places has really bad up speeds, which is what a game server needs the most. You mention "5 players and it starts crapping out" so that might be where you are hitting your up rate limit.

Also, do you really need 60 updates a second?


That's going about the backwards, looking at solutions and then seeing if the problems exist.


Open a profiler, measure, and discover the thing that is actually slow. After the actual problem is discovered, fix that problem. Then measure again.

In real life examples I've seen, I have profiled code to discover that the entire performance problems are based in accidental assumptions. The issues are generally not the problems that were suspected.

In one specific case a very tight loop inside a tool was calling strlen() on every item in a huge list, every single update. The programmer working on the system thought it was a problem with a completely different system, and kept swearing up and down that the other system was as fast as it could go, so the loop was fully optimized. Obviously, opening up every sting and running its full length will be painful on performance. A very minor change to not re-calculate the length dropped this the tool from requiring about five minutes to running almost instantly.


In another specific case, a tool was trying to calculate dependencies on data files within the build system. The existing system had been in use for almost a decade on multiple shipped titles. Various people had mucked with the system, made some improvements buying a few seconds here or there, and moved on. I got sick of it and plugged it in to the profiler to see what was taking so long. Every time it validated a file, it would run a query across the network, see if the file existed on a remote server's disk, and then continued. It ran this query on every item in the game database of several hundred thousand items. People on the team just assumed it was because the game had grown for so many years, it was a legacy system, so that's just how it was. After measuring and finding the actual problems with a profiler, I first fixed remote disk lookup and changed it to a local lookup. This dropped the run time down from about 15 minutes to about 3 minutes. Profiling some more, I could still see a huge amount of the remaining time was spent in OS calls to look up file names; there were thousands of times more lookups than the number of actual files. So I checked the directory tree at startup, cached it in a hash map, and used that to look them up. This dropped the total run time down to about 45 seconds.

In another specific case, processing some line-wrapped text caused the system to grind to a halt. One programmer suspected the font renderer because stopping execution always showed it running in the pre-render step. Another thought it was the font engine calculating font sizes, since it frequently stopped looking up those details. After ACTUALLY PROFILING the system, it was caused by an incredibly stupid algorithm for detecting when to wrap a line of text: The first character would be pre-rendered, then the code would calculate its bounding box and see if it fit. Then the first two characters would be pre-rendered and manually re-calculate the bounding box. Then three characters would be pre-rendered, bounding box figured out, and line wrapped if appropriate. So the worst case of writing out several paragraphs of line-wrapped text caused a thousand or so pre-renders to calculate word wrap by adding one character at a time. To compound it, the results were re-calculated every frame. Fixing the incredibly naive algorithm completely solved that bottleneck.

If those other programmers had simply jumped in and followed their guesses to what was slow, they would not have found and fixed the problems.




Always get a proper diagnosis before prescribing a solution.

Hi

I made a multiplayer 3d game and the server eats up way too much CPU. On a quad-core 3Ghz machine it only supports 5 players before things get realllll slooooooowwww.

The server is multithreaded, one thread per player and checks the line for input 60 times a second.

Does this make sense? Is my design somehow flawed? Does anybody know how I can up the number of players?

I can't check the line less often coz then player movement is disrupted.

Thanks


You answered your own question. You spawn a thread for each player and you only have 4 cores. This wouldn't be a problem if you put a small delay in the threads but sense you just let them run 60 times a second you are going to gobble up all of the CPU memory. I ran into a similar problem with a chat application but a simple thread.sleep for 10 miliseconds changed my server from using 100% CPU to 4%. This is probably still not the optimal way to do this.

Remember to mark someones post as helpful if you found it so.

Journal:

http://www.gamedev.net/blog/908-xxchesters-blog/

Portfolio:

http://www.BrandonMcCulligh.ca

Company:

www.gwnp.ca

This topic is closed to new replies.

Advertisement