Hello,

I am sending compressed json data from the UE4 client to a C++ server made with boost.
I am using ZLib to compress and decompress all json but it doesn't work. I am now encoding it in base64 to avoid some issues but that doesn't change a thing.

I currently stopped trying to send the data and I am writing it in a file from the client and trying to read the file and decompress on the server side.
When the server is trying to decompress it I get an error from ZLib : zlib error: iostream error

My question is the following : Did anyone manage to compress and decompress data between a UE4 client and a C++ server ?
I cannot really configure anything on the server side (because boost has its ZLib compressor) and I don't know what is wrong with the decompression.

Any idea ?

rXp

I know my packet and data receiving/sending is not the problem since when I don't compress everything is transferred fine.
I just have the size as a uint32 header and the rest is the message itself.

When a message is fully received it is sent to my messagemanager to be consume and decompressed. I tested everypart of the client and the server. If I disable the Zlib part everything works. 

Server side compression (the decompression is pretty similar, use zlib_decompressor and read from the right place etc) : 

void MessageManager::compressString(const std::string &data, std::vector<char> &compressedData)
{
    std::stringstream compressed;
    std::stringstream decompressed(data);

    boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
    in.push(boost::iostreams::zlib_compressor());
    in.push(decompressed);
    boost::iostreams::copy(in, compressed);

    std::string str = encode64(decompressed.str());
    compressedData.assign(str.begin(),str.end());
}

Client side compression (it's also pretty easy to understand what the decompress code looks like) : 

void MessageManager::compressString(const FString &json, TArray<uint8> &compressedData)
{
	// Compress File 
	FBufferArchive binaryArrayArchive;
	FString strData = FString(json);
	binaryArrayArchive << strData;
	TArray<uint8> dataArray(binaryArrayArchive.GetData(),binaryArrayArchive.GetAllocatedSize());
	FArchiveSaveCompressedProxy  compressor =
		FArchiveSaveCompressedProxy(compressedData, ECompressionFlags::COMPRESS_ZLIB);
	compressor << dataArray;
	compressor.Flush();
	compressor.FlushCache();
}

If I compress with the server I can decompress with the server. Same with the client.

So now I starting to read UE4 source code and boost source code to see what are the default parameters sent to ZLib, they might differ a lot. I'm also looking into FString of UE4 because std::streamstring might not support the encoding of FString etc...

Oh and yes the encoding in base64 is not necessary at all but I did it since I am making test through files and I wanted to be able to check every character is written in the file.

It is usually wrong i C/C++ to use a string as target for binary data, because strings most often are defined to end at the first occurrence of '\0', but binary data may continue after the first 0. I see the target being a stringstream and know that zlib may contain 0 values, so I expect problems due to the described reason. Some base64 encoding, besides being needless overhead as hplus0603 has already written, does no change anything, because it is applied after the mishap has happened.

I've never used exactly that stuff, so I may be wrong. But I'd definitely try to avoid strings for binary data. Couldn't you copy from the filtering_streambuf to the vector directly?

On the server side I get zlib error: iostream error and the threads stops. It should be an iostream error and only happens when the compress data comes from the client. The client says the header is wrong.

I agree with encore I removed it but std::string can contain \0 it is not delimited by it (although I will look into putting it in a char vector).

I already checked the size on the client, on the server and I checked the byte array data to verify it was the same byte (it is).

The problem is that both header are different. The header from server compressing something is what the RFC of ZLib (ZLib RFC) describe.

But when the client is compressing it, it does not correspond to what it should be, this is the first 2 bytes : C1 83.
I mean it is not a endianess problem since both are running on the same machine and 1C 38 would correspond to the RFC either.
I am looking on how it is compressed by UE4 when it uses ZLib or I will import ZLib myself and do it directly.

Well I got it to work by linking and using the raw zlib library on the UE4 client, I also had to encode into ANSI char for my server to be able to convert it into std::string (I will need to look into utf8 on the server but not for now since I don't send any text message yet).

For those interested I will give my solution for the compression decompression here and then you can tweak it.

Client UE4 :

First of all build the zlib library and I choose the same version as the one in my current version of UE4 (4.18), so I used zlib 1.2.8.

Then I built it manually on windows, but since it's cmake it wont be much different on linux or osx :
 

mkdir C:\Builds\zlib; cd C:\Builds\zlib
cmake -G "Visual Studio 15 2017" -A x64 C:\local\zlib-x.x.x
cmake --build .

Then you need to tell UE4 you want to link it into your project (*.build.cs) :
 

PublicAdditionalLibraries.Add(@"PATHTOPROJECT/Binaries/Win64/zlibd.lib");

(if anyone knows how to have the path to the project dynamically it would be nice)

Compress on the client :

void MessageManager::compressString(const FString &json, TArray<uint8> &compressedData)
{
	//convert into ANSI to be able to cast it into std::string for now
	auto jsonANSI = StringCast<ANSICHAR>(*json);
	TArray<uint8> UncompressedBinaryArray((uint8*)jsonANSI.Get(), jsonANSI.Length());
  
	compressedData.SetNum(UncompressedBinaryArray.Num() * 1023, true);

	//int ret;
	z_stream strm;
	strm.zalloc = Z_NULL;
	strm.zfree = Z_NULL;
	strm.opaque = Z_NULL;

	strm.avail_in = UncompressedBinaryArray.Num();
	strm.next_in = (Bytef *)UncompressedBinaryArray.GetData();
	strm.avail_out = compressedData.Num();
	strm.next_out = (Bytef *)compressedData.GetData();


	// the actual compression work.
	deflateInit(&strm, Z_DEFAULT_COMPRESSION);
	deflate(&strm, Z_FINISH);
	deflateEnd(&strm);

	// Shrink the array to minimum size
	compressedData.RemoveAt(strm.total_out, compressedData.Num() - strm.total_out, true);
}

Decompress on the client :

FString MessageManager::decompressString(TArray<uint8> &compressedData)
{
	TArray<uint8> UncompressedBinaryArray;
	UncompressedBinaryArray.SetNum(compressedData.Num() * 1032);

	//int ret;
	z_stream strm;
	strm.zalloc = Z_NULL;
	strm.zfree = Z_NULL;
	strm.opaque = Z_NULL;

	strm.avail_in = compressedData.Num();
	strm.next_in = (Bytef *)compressedData.GetData();
	strm.avail_out = UncompressedBinaryArray.Num();
	strm.next_out = (Bytef *)UncompressedBinaryArray.GetData();

	// the actual DE-compression work.
	inflateInit(&strm);
	inflate(&strm, Z_FINISH);
	inflateEnd(&strm);

	return FString((ANSICHAR*)UncompressedBinaryArray.GetData());
}

On the server (C++ using boost and STD) this is how to compress :

void MessageManager::compressString(const std::string &data, std::vector<char> &compressedData)
{
    std::stringstream compressed;
    std::stringstream decompressed(data);

    boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
    in.push(boost::iostreams::zlib_compressor());
    in.push(decompressed);
    boost::iostreams::copy(in, compressed);

    std::string str = compressed.str();
    compressedData.assign(str.begin(),str.end());
}

And decompress :

std::string MessageManager::decompressString(std::vector<char> &compressedData)
{
    std::stringstream compressed;
    compressed.write(compressedData.data(),compressedData.size()); // <--- that is to be sure that it WONT STOP copying at a \0 char
    std::stringstream decompressed;

    boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
    in.push(boost::iostreams::zlib_decompressor());
    in.push(compressed);
    boost::iostreams::copy(in, decompressed);

    std::string str(decompressed.str());
    return str;
}

I hope it will help someone out there

13 hours ago, hplus0603 said:

 

 

Good you got it working, but that's till doing it wrong.

std::string is not for binary data. You should be using std::vector<char> or some other binary buffer format, rather than std::string. Encoding to ASCII neutralizes the entire idea of compression, because it expands the representation of the data again before it goes over the wire.

 

I encoded into ansi before the compression not after.