• FEATURED

View more

View more

View more

### Image of the Day Submit

IOTD | Top Screenshots

### The latest, straight to your Inbox.

Subscribe to GameDev.net Direct to receive the latest updates and exclusive content.

# Your most valuable debugging techniques for Networked games?

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

16 replies to this topic

### #1JordanBonser  Members

Posted 23 April 2014 - 03:26 AM

POPULAR

I have recently started adding networking to a game I have been making but the development of this in comparison to a single player game seems extremely slow. This is mostly due to the complexity of debugging issues that occur. As a general rule the problem can be:

• sending the wrong packet data to the server.
• the server unpackaging this data incorrectly.
• the server sending the wrong data to the clients.
• the server applying the data in the wrong way.
• the clients unpackaging the data incorrectly.
• the clients applying the data in the wrong way.

So this can get quite complex and the main ways I have been debugging so far is through either trying to debug normally in the IDE. Outputting a lot of Logging information, which can be tedious. Outputting some debug information to the screen which is often fairly limited but can show problems in real time.

I'm just wondering if anyone else has any other better ideas or any network debugging tips that have really helped them?

### #2Ashaman73  Members

Posted 23 April 2014 - 03:50 AM

POPULAR

I would focus on the log-file. How about writing a log-file analyser,which looks out for certain errors, check protocol messages, compares protocol server msg with client msg etc. It could be a valuable investment.

Ashaman

### #3JordanBonser  Members

Posted 23 April 2014 - 04:40 AM

I would focus on the log-file. How about writing a log-file analyser,which looks out for certain errors, check protocol messages, compares protocol server msg with client msg etc. It could be a valuable investment.

Ahh yeah, I never really thought of making it that descriptive. So far I have literally just output certain values, like the values of a matrix and checked them by eye when I have needed to. So to expand a bit, you would recommend outputting the values of the packets before sending and when recieved to a log file.Then create a small program to scan the file and check the values of the packets, throwing an error if any differentiate?

### #4Ashaman73  Members

Posted 23 April 2014 - 06:11 AM

Then create a small program to scan the file and check the values of the packets, throwing an error if any differentiate?

Yes, this is one option. More important are protocol msg order, eg

Client 1 request game slot
Client 2 request game slot
Client 1 assign slot 3
Client 1 sends user data
Client 2 assign slot 4
Client 2 sends user data
Client 1 accepted user
Client 2 denied user
Client 1 request level data
Client 1 sends join signal
Client 2 sends join signal
<crash....>

One option would be a filter like this

Client 1 msg:
- request game slot
- assign slot 3
- sends user data
- accepted user
- request level data
- sends join signal


An second option would be analysing the protocol:

Client 2 sends join signal after being denied !!


Which could be the cause of the crash.

Ashaman

### #5nfactorial  Members

Posted 23 April 2014 - 06:13 AM

You should have unit tests for all the issues you list. That would eliminate packing/unpacking/applying from your list immediately at the very least.

n!

### #6JordanBonser  Members

Posted 23 April 2014 - 06:36 AM

Ahh yes that does seem to be a much better way of interpreting the data and finding the cause of crashes, thanks a lot Yeah I should definitely write unit tests for all my packet classes, another great tip!

Is it worthwhile do you think to have some sort of GUI for checking the state of objects in the scene? maybe like a console with special commands?

### #7Angus Hollands  Members

Posted 23 April 2014 - 06:58 AM

Unit tests are a god send!

I've never been exposed to their necessity in a production environment, but now I don't think that I will need that experience to encourage me to use them. If you can't write a unit test for it, it probably needs refactoring!

Most network errors are caused by misreading data from the byte stream (off by-n errors).I always check the data coming in. However, recently I had an error in some reading code which caused an off by one error. The best thing to do in those circumstances (my error only arose after data was already sent) is to break the simulation into three different states and compare the transition between them (whenever my variable was set to None, it remained considered as a None value (Which isn't included in the byte stream) hence we forgot to read an extra byte from later packets! I had to break it down to fix it.

Having an inspector in game is really useful, (assuming you're a runtime interpreted language) even a GUI with a command input dialogue is all you'd need (as long as you can prevent the game state from progressing)

### #80BZEN  Members

Posted 23 April 2014 - 07:26 AM

logging everything.

Everything is better with Metal.

### #9Glass_Knife  Moderators

Posted 23 April 2014 - 07:45 AM

POPULAR

1.  Rolling log files that keep a certain amount of data (10 Mb) or so you always have enough to go back and find the problem.  If it would help, you can log data in XML or JSON or some kind of parsable format so you can generate HTML views of the data or sort through it with code.  This can help in with finding problems you didn't know were there.

2.  Wireshark - Some kind of network protocol analyzer can really help with weird bugs.  You won't need it often, but sometimes this is the only way to figure out what is happening.

3.  Testing - there should be a clear separation between the networking code and the rest of the system.  I always have a Network interface or something like that.  You can create a mock network for testing that simulates all the weird things that can happen: packets out of order, lost data, etc.  If you assume that the network code works, you can test all the code that uses the network to verify it behaves as expected.  It isn't possible to test all the network functionality, but if you test that your code gives the correct stuff to the network class, and anything taken from the network class is handled correctly, then there is a lot less that can go wrong.  And when you find that the server behaves in a weird way, you can test that your classes respond correctly to the new buggy behavior you've discovered.

Edited by Glass_Knife, 23 April 2014 - 07:46 AM.

I think, therefore I am. I think? - "George Carlin"
My Website: Indie Game Programming

My Book: http://amzn.com/1305076532

### #10hplus0603  Moderators

Posted 23 April 2014 - 09:52 AM

POPULAR

but the development of this in comparison to a single player game seems extremely slow

Yes, it is, because a distributed system is a lot more complex than a self-contained system!

And it will keep getting slower, because you haven't yet brought up issues where "player A sees something different from player B" or "this problem only happens when that particular packet doesn't make it to some of the players" or "timing."

I agree with the other posters -- keeping a log of all packets sent and received since start-up is a good way to debug, and unless your server stays up for days on end, you should have ample disk space to keep the entire log around.

Also, once you have the "converting packets to structures and back" marshaling part of networking down in a re-usable library, that won't really be a problem anymore. That's when the real challenge of distributed computing starts!
enum Bool { True, False, FileNotFound };

### #11C0lumbo  Members

Posted 23 April 2014 - 11:49 AM

Make sure that your game launching code lets you create multiplayer games with just one player in debug mode, make it so that it uses as much of the regular multiplayer logic as possible.

Put systems in place to allow creation of AI driven clients (ideally running in a console window on the same machine, so you're not wasting time with multiple machines)

### #12wodinoneeye  Members

Posted 28 April 2014 - 07:49 PM

Logging to a file and simultaneously logging to a seperate visible logging window.

Add a small library of logging tools to make it very easy to add logging message calls (including major/minor indicators in the logs to make important messages stand out, and useful things like imbedded timestamps, dumping of variable values, socket state blocks, etc..)

Having a way to subset messaging to turn off logging of 'priority' sets to lower that mess of log data to look thru  (sometimes you want to turn off the loging for ordinary stuff , but once something goes wrong you want it all back again and adding/removing logging message calls over and over is wasteful)

Dont forget to flush you logger (file) every message (mine usually opens/closes the file every message)  and with threading usually the case adding the protects for the read/write operations...

--------------------------------------------Ratings are Opinion, not Fact

### #13JordanBonser  Members

Posted 29 April 2014 - 01:47 AM

There has been some great tips, I think I will need to upgrade my logging classes quite a lot. I particularly like the idea wodinoneeye had about having different priorities of logging. I'm thinking of making it possible to change this in real time as well so although there will still be one large log file saved, I will also have another log file that will only apply for certain types of log messages.

### #140BZEN  Members

Posted 29 April 2014 - 07:37 AM

POPULAR

- Make sure you also do some unit testing.

More stuff, maybe :

- Internet conditions simulation. Either basic, for a quick test, or pro-level stuff, with dedicated hardware. Will not replace real-world conditions but I find it always useful.

- Since you're dealing with binary data (most likely), some form of checksum / type-checking / range-checking verification. This mostly applies to debugging things like serialisation / deserialisation and marshaling.

- Some form of binary -> readable text conversion on packets. Usually packets are tightly packed binary streams and virtually unreadable. So, some form of analyzing packet content can be useful if you run into trouble with your packets turning up as garbage.

- Recording packets, game states, inputs, and replay features. Pretty high level, and a lot of work in itself, so you may want to skip that.

Checksum and replay especially useful if you have the intention of running a deterministic lock-step engine.

At a game level :

- A basic versioning system. If your packet / message format isn't backward compatible, you'll end up with garbage data.

- Dependencies check. Making sure whoever connects to your server has all the custom content it needs to being able to play the game. Either the right version, the map data, ect...

- A replay system can also help you catching strange game exploits. And if your game supports replays, then why not

Edited by 0BZEN, 29 April 2014 - 07:42 AM.

Everything is better with Metal.

### #15snacktime  Members

Posted 24 May 2014 - 05:13 AM

Unit testing is the only sane way to handle this.  While integration testing is popular in the industry, there is some fairly compelling data out there that shows that the effectiveness of integration testing goes down exponentially the more complex the system gets as a whole.  My own experience tends to concur.

Especially for networked applications, I am a strong proponent of using a lot of mocking and stubbing.   You want to keep your tests isolated to what you want to test, not all the other stuff.

Unit testing is also a good incentive to write better abstractions.  Good abstractions are just  easier to test, and you will find yourself just writing better overall code when you start unit testing.

### #16Angus Hollands  Members

Posted 24 May 2014 - 08:05 AM

A follow up from my post.
I've been changing a lot of serialiser code and I have my unit tests run whenever T import my library (for debugging purposes, of course). It's caught bugs that I'd have to labour over otherwise!

### #17Kylotan  Moderators

Posted 10 June 2014 - 06:16 AM

I may as well chime in with some late observations.

• sending the wrong packet data to the server.
• the server unpackaging this data incorrectly.

As already mentioned, these 2 are perfect candidates for unit testing. Every single part of your serialisation and deserialisation code should be trivial to write tests for and therefore you can be in a position where you are strongly confident that there are no bugs here.

Another idea I like to implement in debug builds is to immediately unpack any packet I'm about to send and verify that the unpacked version matches the packed version, using an assert. This catches several types of serialisation bug before the data even hits the wire.

• the server sending the wrong data to the clients.

Obviously this would be a general logic error, but one way to reduce this problem is to limit the variety of data that can be sent. Some games create hundreds of different packets, one for every piece of functionality. If you can reduce these disparate messages into a much smaller set, then there is less scope for error. My MMO code has no more than roughly 10 distinct message types, and although each of those has some subtypes, they all pass through the same few basic functions and are easy to check for errors.

• the server applying the data in the wrong way.

Again, this is a general logic error, no different from doing the wrong thing in any code. But if you have reduced the number of distinct messages then it would seem harder to do that.

• the clients unpackaging the data incorrectly.
• the clients applying the data in the wrong way.

Ideally your clients and servers use the same code, so if you fix the problem at one end, you'll fix it at both. But if you can't share the same code - and I know what this is like, having worked on an MMO with a Python server but clients written in C# and C++ - just try to keep the code as similar as possible and unit test as much as you can.

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.