C++/Python Compression
Author
I'm looking for a compression library that I can use with both Python and C++. I need to be able to compress files with a tool I am writing in Python, then decompress them in my game, which is written in C++. I was looking at zlib, but I'm not sure if the C++ zlib library uses the same algorithm as the Python zlib library. Does anyone have any suggestions?
Quote:Original post by Eddy999999
I'm not sure if the C++ zlib library uses the same algorithm as the Python zlib library.
It does.
Author
Well, I got it working in my Python program, but I'm having some trouble figuring out how to get zlib working in my C++ program. I'd like to have a function where I can just say something like:
But the closest I've found to that is this, which takes it's input and output to/from files, instead of strings, and it won't work for me anyways. When I try to compile the sample program, I get a bunch of "unresolved external symbol" errors (I made sure I had the right headers and libraries included).
If anyone could help me out, it would be greatly appreciated.
decompressedString=decompress(compressedString);But the closest I've found to that is this, which takes it's input and output to/from files, instead of strings, and it won't work for me anyways. When I try to compile the sample program, I get a bunch of "unresolved external symbol" errors (I made sure I had the right headers and libraries included).
If anyone could help me out, it would be greatly appreciated.
Author
I got the example working, and since noone has offered any input, I figured I'd try to mess with it to try to make it read/write strings instead of files. To start with, I'm trying to make the compression function compress a string into a file. Here's the original compression function:
I'm not too familiar with streams and such, but I'm thinking that I just need to make fin contain the string I want, rather than the data from a file. However, I believe it still has to be the FILE type. So I tried changing
to
Which compiles and runs fine, but it doesn't give the expected output. After the compression function runs, and the file is decompressed again, instead of containing "teststring...", the output file contains about 4kb of the character Í. Once again, I don't know much about streams or anything, so if anyone could please help me out, it would be greatly appreciated.
Also, if anyone needs the full code for the ZlibEngine class, I can post it (or you can download it from the page I posted above).
int ZlibEngine::compress( const char *input, const char *output, int level ){ err = Z_OK; avail_in = 0; avail_out = output_length; next_out = output_buffer; m_AbortFlag = 0; fin = fopen( input, "rb" ); fout = fopen( output, "wb" ); length = filelength( fileno( fin ) ); deflateInit( this, level ); for ( ; ; ) { if ( m_AbortFlag ) break; if ( !load_input() ) break; err = deflate( this, Z_NO_FLUSH ); flush_output(); if ( err != Z_OK ) break; progress( percent() ); } for ( ; ; ) { if ( m_AbortFlag ) break; err = deflate( this, Z_FINISH ); if ( !flush_output() ) break; if ( err != Z_OK ) break; } progress( percent() ); deflateEnd( this ); if ( m_AbortFlag ) status( "User Abort" ); else if ( err != Z_OK && err != Z_STREAM_END ) status( "Zlib Error" ); else { status( "Success" ); err = Z_OK; } fclose( fin ); fclose( fout ); fin = 0; fout = 0; if ( m_AbortFlag ) return Z_USER_ABORT; else return err;}I'm not too familiar with streams and such, but I'm thinking that I just need to make fin contain the string I want, rather than the data from a file. However, I believe it still has to be the FILE type. So I tried changing
fin = fopen( input, "rb" );to
fin=tmpfile();fputs("teststring...",fin);Which compiles and runs fine, but it doesn't give the expected output. After the compression function runs, and the file is decompressed again, instead of containing "teststring...", the output file contains about 4kb of the character Í. Once again, I don't know much about streams or anything, so if anyone could please help me out, it would be greatly appreciated.
Also, if anyone needs the full code for the ZlibEngine class, I can post it (or you can download it from the page I posted above).
Why are you using files, if all you want is to compress/decompress from one memory location to another ? That's horribly inefficient.
zlib specifically offers functions to compress / decompress from and to memory:
and
zlib specifically offers functions to compress / decompress from and to memory:
Quote:
ZEXTERN int ZEXPORT compress2 OF((Bytef *dest, uLongf *destLen,
const Bytef *source, uLong sourceLen,
int level));
/*
Compresses the source buffer into the destination buffer. The level
parameter has the same meaning as in deflateInit. sourceLen is the byte
length of the source buffer. Upon entry, destLen is the total size of the
destination buffer, which must be at least the value returned by
compressBound(sourceLen). Upon exit, destLen is the actual size of the
compressed buffer.
compress2 returns Z_OK if success, Z_MEM_ERROR if there was not enough
memory, Z_BUF_ERROR if there was not enough room in the output buffer,
Z_STREAM_ERROR if the level parameter is invalid.
*/
and
Quote:
ZEXTERN int ZEXPORT uncompress OF((Bytef *dest, uLongf *destLen,
const Bytef *source, uLong sourceLen));
/*
Decompresses the source buffer into the destination buffer. sourceLen is
the byte length of the source buffer. Upon entry, destLen is the total
size of the destination buffer, which must be large enough to hold the
entire uncompressed data. (The size of the uncompressed data must have
been saved previously by the compressor and transmitted to the decompressor
by some mechanism outside the scope of this compression library.)
Upon exit, destLen is the actual size of the compressed buffer.
This function can be used to decompress a whole file at once if the
input file is mmap'ed.
uncompress returns Z_OK if success, Z_MEM_ERROR if there was not
enough memory, Z_BUF_ERROR if there was not enough room in the output
buffer, or Z_DATA_ERROR if the input data was corrupted or incomplete.
*/
Author
I'm not trying to deal with files, just the only example I could find dealt with files, so I was trying to work from there. I saw the functions you posted in the manual, but as I said above, I don't really know much about streams, so I have no idea how to actually use the functions. I also don't know what most of those data types mean. For example, I know what bytes and long integers are, but what's special about the Bytef and uLongF data types.
I guess I just need some samples to look at to fully understand exactly how to actually use the functions, because right now it's Greek to me, since I simply don't understand a lot of the terms/data types used.
P.S. And before you try to say I'm lazy and just want to be spoon fed, I've been working on this pretty much all day, and part of last night, searching google, reading through documentation, etc. and so far it just hasn't clicked...
EDIT: After reading through the documentation a bit more, it's starting to make a bit more sense. The biggest question I have now is what is the Bytef datatype? I'm gathering it's a stream of some type, but what's different about it? Also, how can I convert a string to a Bytef and vice versa?
[Edited by - Eddy999999 on March 2, 2008 10:15:21 PM]
I guess I just need some samples to look at to fully understand exactly how to actually use the functions, because right now it's Greek to me, since I simply don't understand a lot of the terms/data types used.
P.S. And before you try to say I'm lazy and just want to be spoon fed, I've been working on this pretty much all day, and part of last night, searching google, reading through documentation, etc. and so far it just hasn't clicked...
EDIT: After reading through the documentation a bit more, it's starting to make a bit more sense. The biggest question I have now is what is the Bytef datatype? I'm gathering it's a stream of some type, but what's different about it? Also, how can I convert a string to a Bytef and vice versa?
[Edited by - Eddy999999 on March 2, 2008 10:15:21 PM]
Quote:Original post by Eddy999999
I'm not trying to deal with files, just the only example I could find dealt with files, so I was trying to work from there. I saw the functions you posted in the manual, but as I said above, I don't really know much about streams, so I have no idea how to actually use the functions. I also don't know what most of those data types mean. For example, I know what bytes and long integers are, but what's special about the Bytef and uLongF data types.
There's nothing really difficult about these functions. They essentially compress or uncompress raw data from one memory location to another location.
The compress function, for example, takes a pointer to your uncompressed data, it's length, and a pointer to where you would like the compressed data to be stored. It returns the length of the compressed data, and an error code.
The strange data types are internally declared by zlib (in zconf.h). ZLib is an extremely portable library, that can run on anything from a mobile phone to a supercomputer. Not all architectures have the same data type sizes, so in order to keep a consistent interface across all platforms, zlib had to do some typedef magic.
You shouldn't really worry about it. For all practical purposes, on an x86 or x64 PC, Bytef is an unsigned char (8 bit), and uLong + uLongf are both unsigned int (32 bit).
To compress a block of data, you can use something like this (from the top of my head, might not compile):
unsigned long CompressData(const char *source, unsigned int sourceSize, char *destination, int level){ // This will hold the size of the compressed data unsigned long dsize; // Compress everything from source to destination int result = compress2((unsigned char *)destination, &dsize, (const unsigned char *)source, sourceSize, level); // Check result for errors if( result != Z_OK ) ... do something ... // Return the size of the compressed data return( dsize );}Keep in mind that the destination memory needs to be large enough to hold the compressed result. You can use the compressBound function to estimate that size prior to compression.
Quote:Original post by Eddy999999
Also, how can I convert a string to a Bytef and vice versa?
Since a Bytef* is just a char pointer, you can use c_str() to convert a string to it, and the std::string constructor (taking a const char pointer) to get it back into a string. Be careful about allocating a large enough memory block to hold the compressed or uncompressed destination data, though (as mentioned above). You can release this memory once you converted it to an std::string.
Author
Well, it's almost working. However, when I try to decompress the data that I've compressed, it comes out with the original string followed by a bunch of "╠" characters, and it gives a Z_DATA_ERROR. Here's the code:
//Compress program#include "zlib.h"#include <stdio.h>#include <string>#include <iostream>#include <fstream>unsigned long CompressData(const char *source, unsigned int sourceSize, char *destination){ // This will hold the size of the compressed data unsigned long dsize; // Compress everything from source to destination int result = compress((unsigned char *)destination, &dsize, (const unsigned char *)source, sourceSize); // Check result for errors if( result != Z_OK ) { std::cout << "Error occured!"; } // ... do something ... // Return the size of the compressed data return( dsize );}int main(int argc, char *argv[]) { char *src="test string..."; char dest[1024]; std::cout << "Compressing string " << src << "\n"; CompressData(src,strlen(src),(char*)&dest); std::cout << dest;}//Decompress program#include "zlib.h"#include <stdio.h>#include <string>#include <iostream>#include <fstream>unsigned long DecompressData(const char *source, unsigned int sourceSize, char *destination){ // This will hold the size of the compressed data unsigned long dsize; // Compress everything from source to destination int result = uncompress((unsigned char *)destination, &dsize, (const unsigned char *)source, sourceSize); // Check result for errors if( result != Z_OK ) { std::cout << "Error occured! Error code: " << result << "\n"; } // ... do something ... // Return the size of the compressed data return( dsize );}int main(int argc, char *argv[]) { char *src="test string..."; char dest[1024]; std::cout << "Decompressing string " << src << "\n"; DecompressData(src,strlen(src),(char*)&dest); std::cout << dest;}This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
