Sign in to follow this  
JordanBonser

Your most valuable debugging techniques for Networked games?

Recommended Posts

JordanBonser    839

I would focus on the log-file. How about writing a log-file analyser,which looks out for certain errors, check protocol messages, compares protocol server msg with client msg etc. It could be a valuable investment. 

Ahh yeah, I never really thought of making it that descriptive. So far I have literally just output certain values, like the values of a matrix and checked them by eye when I have needed to. So to expand a bit, you would recommend outputting the values of the packets before sending and when recieved to a log file.Then create a small program to scan the file and check the values of the packets, throwing an error if any differentiate? 

Share this post


Link to post
Share on other sites
Ashaman73    13715


Then create a small program to scan the file and check the values of the packets, throwing an error if any differentiate?

Yes, this is one option. More important are protocol msg order, eg

Client 1 request game slot
Client 2 request game slot
Client 1 assign slot 3
Client 1 sends user data
Client 2 assign slot 4
Client 2 sends user data
Client 1 accepted user
Client 2 denied user
Client 1 request level data
Client 1 sends join signal
Client 2 sends join signal
<crash....>

One option would be a filter like this

Client 1 msg:
- request game slot
- assign slot 3
- sends user data
- accepted user
- request level data
- sends join signal

An second option would be analysing the protocol:

Client 2 sends join signal after being denied !!

Which could be the cause of the crash.

Share this post


Link to post
Share on other sites
JordanBonser    839

Ahh yes that does seem to be a much better way of interpreting the data and finding the cause of crashes, thanks a lot :) Yeah I should definitely write unit tests for all my packet classes, another great tip!

 

Is it worthwhile do you think to have some sort of GUI for checking the state of objects in the scene? maybe like a console with special commands? 

Share this post


Link to post
Share on other sites
Angus Hollands    869

Unit tests are a god send!

 

I've never been exposed to their necessity in a production environment, but now I don't think that I will need that experience to encourage me to use them. If you can't write a unit test for it, it probably needs refactoring!

 

Most network errors are caused by misreading data from the byte stream (off by-n errors).I always check the data coming in. However, recently I had an error in some reading code which caused an off by one error. The best thing to do in those circumstances (my error only arose after data was already sent) is to break the simulation into three different states and compare the transition between them (whenever my variable was set to None, it remained considered as a None value (Which isn't included in the byte stream) hence we forgot to read an extra byte from later packets! I had to break it down to fix it.

 

Having an inspector in game is really useful, (assuming you're a runtime interpreted language) even a GUI with a command input dialogue is all you'd need (as long as you can prevent the game state from progressing)

Share this post


Link to post
Share on other sites
C0lumbo    4411

Make sure that your game launching code lets you create multiplayer games with just one player in debug mode, make it so that it uses as much of the regular multiplayer logic as possible.

Put systems in place to allow creation of AI driven clients (ideally running in a console window on the same machine, so you're not wasting time with multiple machines)

Share this post


Link to post
Share on other sites
wodinoneeye    1689

Logging to a file and simultaneously logging to a seperate visible logging window.

Add a small library of logging tools to make it very easy to add logging message calls (including major/minor indicators in the logs to make important messages stand out, and useful things like imbedded timestamps, dumping of variable values, socket state blocks, etc..)  

 

Having a way to subset messaging to turn off logging of 'priority' sets to lower that mess of log data to look thru  (sometimes you want to turn off the loging for ordinary stuff , but once something goes wrong you want it all back again and adding/removing logging message calls over and over is wasteful)

 

Dont forget to flush you logger (file) every message (mine usually opens/closes the file every message)  and with threading usually the case adding the protects for the read/write operations...

Share this post


Link to post
Share on other sites
JordanBonser    839

There has been some great tips, I think I will need to upgrade my logging classes quite a lot. I particularly like the idea wodinoneeye had about having different priorities of logging. I'm thinking of making it possible to change this in real time as well so although there will still be one large log file saved, I will also have another log file that will only apply for certain types of log messages.

Share this post


Link to post
Share on other sites
snacktime    452

Unit testing is the only sane way to handle this.  While integration testing is popular in the industry, there is some fairly compelling data out there that shows that the effectiveness of integration testing goes down exponentially the more complex the system gets as a whole.  My own experience tends to concur.

 

Especially for networked applications, I am a strong proponent of using a lot of mocking and stubbing.   You want to keep your tests isolated to what you want to test, not all the other stuff.

 

Unit testing is also a good incentive to write better abstractions.  Good abstractions are just  easier to test, and you will find yourself just writing better overall code when you start unit testing.

Share this post


Link to post
Share on other sites
Angus Hollands    869

A follow up from my post.
I've been changing a lot of serialiser code and I have my unit tests run whenever T import my library (for debugging purposes, of course). It's caught bugs that I'd have to labour over otherwise!

Share this post


Link to post
Share on other sites
Kylotan    9860

I may as well chime in with some late observations.

  • sending the wrong packet data to the server.
  • the server unpackaging this data incorrectly.

As already mentioned, these 2 are perfect candidates for unit testing. Every single part of your serialisation and deserialisation code should be trivial to write tests for and therefore you can be in a position where you are strongly confident that there are no bugs here.

 

Another idea I like to implement in debug builds is to immediately unpack any packet I'm about to send and verify that the unpacked version matches the packed version, using an assert. This catches several types of serialisation bug before the data even hits the wire.

  • the server sending the wrong data to the clients.

Obviously this would be a general logic error, but one way to reduce this problem is to limit the variety of data that can be sent. Some games create hundreds of different packets, one for every piece of functionality. If you can reduce these disparate messages into a much smaller set, then there is less scope for error. My MMO code has no more than roughly 10 distinct message types, and although each of those has some subtypes, they all pass through the same few basic functions and are easy to check for errors.

  • the server applying the data in the wrong way.

Again, this is a general logic error, no different from doing the wrong thing in any code. But if you have reduced the number of distinct messages then it would seem harder to do that.

  • the clients unpackaging the data incorrectly.
  • the clients applying the data in the wrong way.

Ideally your clients and servers use the same code, so if you fix the problem at one end, you'll fix it at both. But if you can't share the same code - and I know what this is like, having worked on an MMO with a Python server but clients written in C# and C++ - just try to keep the code as similar as possible and unit test as much as you can.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this