Hello, I am working on a graphical MUD and have noticed that I am currently using a lot of bandwidth. I do ofcourse plan to reduce this by being more restrictive with what I send, but I have also considered to start compressing the data before sending it. My question is, how should I do this to make it work good. My first Idea is that I should put everything I want to send into a buffer, and by a certain interval (25 ms or so) I will compress and send everything that is in the buffer at the time. Is this a good way to do it?

There are a couple of warning flags in your post. It sounds like you're currently not buffering data... which could imply that you'd be hammering the bandwidth with per-packet overhead for TCP/UDP headers if you send lots of small bits of data individually. (Which protocol are you using, incidentally? And language/library?) That alone could be a big waste of your bandwidth. Do you not have a predefined message format that you use to send your data? If so, why not just compress the contents of each message individually? And if not, what are you using instead?

[Edited by - Kylotan on March 29, 2010 9:00:57 AM]

I am using TCP with winsock2.

Well, I have a predefined message format so that I can collect all data needed for a certain operation into one buffer and send it when it is ready, and the other side will know how much data to receive. So if a client requests to move it's character the server gather information about the map, units and objects and then call send(), and the client will then know how much data will be incoming and which parts will contain which information without first receiving information about it. There are ofcourse exceptions where the amount of data must be different, e.g. depending on how many characters that are within range.


My point is that certain operations, like updating a character's HP require kind little data to be transferred. I am already merging this kind of calls together before sending them, so that every 50 ms I send one update that includes new information about every character that has changed, if any change has occured.
But I am considering using this method for all communication, so that I more effectivly can compress the data using a library like zlib.

The compression libraries I have worked with usually compress when writing to the outgoing buffer:

buffer.WriteInt32(value)
buffer.WriteInt16(value)

etc.

Are there any compression libraries out there capable of handling this type of streaming in a reasonable way? Because I'm fairly certain that the usual suspects (zlib, bzip2, lzma, etc) would force you to compress individual packets.

In theory it should be straightforward to modify a LZ77 coder to see the earlier packets, or an adaptive arithmetic coder to flush its output buffer at the end of a message, and still get excellent compression rates.

Quote:Original post by Exustio
Hello, I am working on a graphical MUD and have noticed that I am currently using a lot of bandwidth. I do ofcourse plan to reduce this by being more restrictive with what I send, but I have also considered to start compressing the data before sending it.

My question is, how should I do this to make it work good.
My first Idea is that I should put everything I want to send into a buffer, and by a certain interval (25 ms or so) I will compress and send everything that is in the buffer at the time. Is this a good way to do it?

I have SendCompressedBuffer and ReceiveCompressedBuffer classes that use bzip2 to compress data. After working with some encryption libraries, I highly recommend bzip2 as it's API/documentation/... are easy to work with and work as advertised.


Brian Wood
http://webEbenezer.net
(651) 251-9384

zlib supports streamed compression too.
I use TCP with disabled nagle, buffer the messages myself and compress them packet-wish with zlib (fastest provides the best ratio for my usecase). And i use a homebrewed library for determinate which data has to be sent. It replaces the 'server-redirect-messages-to-clients'-approach by a 'server-knpws-what-a-client-know-determinates-changes-and-visibilty-and-sends-only-important-data'-approach.
This combination or at least some parts may be interesting for you either:
http://syncsys.sourceforge.net/ (opensource of course)

Quote:Original post by catch
25ms seems far too frequent for a mostly-event based game like a MUD.

For a MUD? I would think 40 network updates per second is more than any mainstream game should attempt, regardless of the genre. Unless he can guarantee a Ethernet connection, that is way too much traffic.


Compression alone will not help a design that is fundamentally flawed.



The OP should figure out his actual bandwidth requirements, including overhead. A bad ratio here can kill any game. The solution there is to send fewer packets. If the ratio is fine but the total bandwidth is an issue, then he should first consider a more efficient data structure, and then consider compression as a secondary possible improvement.

The OP should also evaluate the effect of latency in his engine compared against the number of updates. What happens if there is some network instability and those 25ms updates get delayed for 3-5 seconds? If it takes a while to catch up, that could seriously impact the game.

Those answers would help with the questions about what to do next.

Given that we're talking about TCP here, my suspicion is just that the server is trying to generate far too much data for some reason. Packet overhead probably isn't that big of a deal - if it's being sent regularly, it'll be getting buffered, and if it's being sent infrequently, the bandwidth wouldn't be that high. Of course, the original poster's bandwidth measurement could be wrong, too. Either way, I think compression is the wrong approach at this point. Some real figures on message size and frequency would help.