Sign in to follow this  
Galapaegos

zlib issue

Recommended Posts

Galapaegos    277
Hi Folks, Our file format uses zlib to compress/decompress files. Essentially, I'm trying to write a new compress/decompress routine to add my own progress meter. I have two test functions, one that loads a file and compresses it reading via 'fread/fwrite' calls, and another function that compresses by using memcpy. The memory mapping function doesn't return the same size as the other method. I need to get the memory mapping function to work, since this is the situation that is happening in my loading function. Here are the two functions for reference: file:
FILE *fin = fopen (arg.c_str (), "rb");
FILE *fout = fopen (out.c_str (), "wb");

//z_stream
z_stream stream;
memset (&stream, 0, sizeof (stream));

int err = deflateInit (&stream, 6);
if (err != Z_OK)
	printf ("zlib:%s\n", stream.msg);

int input_length = MAGIC_NUMBER;
unsigned char input_buffer [MAGIC_NUMBER];
int output_length = MAGIC_NUMBER;
unsigned char output_buffer [MAGIC_NUMBER];

stream.next_in = input_buffer;
stream.next_out = output_buffer;

stream.avail_in = fread (input_buffer, 1, input_length, fin);
stream.avail_out = MAGIC_NUMBER;

int read = 1, bytescompressed = MAGIC_NUMBER, decompressed = 0;

for (; ;)
{
	if (stream.avail_in == 0)
	{
		stream.avail_in = fread (input_buffer, 1, input_length, fin);
		stream.next_in = input_buffer;

		read ++;
	}

	err = deflate (&stream, Z_NO_FLUSH);

	unsigned long count = output_length - stream.avail_out;
	if (count)
	{
		if (fwrite (output_buffer, 1, count, fout) != count)
		{
			err = Z_ERRNO;
			return err;
		}

		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;
	}

	if (err != Z_OK)
		break;
}

for (; ;)
{
	err = deflate (&stream, Z_FINISH);
	
	unsigned int count = output_length - stream.avail_out;
	if (count)
	{
		if (fwrite (output_buffer, 1, count, fout) != count)
		{
			err = Z_ERRNO;
			return err;
		}

		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;
	}

	if (err != Z_OK)
		break;
}

deflateEnd (&stream);
if ((err != Z_OK) && (err != Z_STREAM_END))
{
	printf ("zlib:%i, %s\n", err, stream.msg);
	inflateEnd (&stream);
	return err;
}

printf ("decompressed:%i\n", decompressed);

fclose (fin);
fclose (fout);

memory mapped:
FILE *fin = fopen (arg.c_str (), "rb");
FILE *fout = fopen (out.c_str (), "wb");

int length = filelength (fileno (fin));
int destLength = length*1.001 + 12;

unsigned char *src = new unsigned char [length];
unsigned char *dst = new unsigned char [destLength];

fread (src, 1, length, fin);

//z_stream
z_stream stream;
memset (&stream, 0, sizeof (stream));

int err = deflateInit (&stream, 6);
if (err != Z_OK)
	printf ("zlib:%s\n", stream.msg);

int input_length = MAGIC_NUMBER;
unsigned char input_buffer [MAGIC_NUMBER];
int output_length = MAGIC_NUMBER;
unsigned char output_buffer [MAGIC_NUMBER];

memset (input_buffer, 0, MAGIC_NUMBER);
memset (output_buffer, 0, MAGIC_NUMBER);

stream.next_in = input_buffer;
stream.next_out = output_buffer;

memcpy (input_buffer, src, MAGIC_NUMBER);
stream.avail_in = MAGIC_NUMBER;
stream.avail_out = MAGIC_NUMBER;

int read = 1, bytescompressed = MAGIC_NUMBER, decompressed = 0;

for (; ;)
{
	if (stream.avail_in == 0)
	{
		memset (input_buffer, 0, MAGIC_NUMBER);

		if ((read+1)*MAGIC_NUMBER < length)
		{
			memcpy (input_buffer, src + read*MAGIC_NUMBER, MAGIC_NUMBER);
			stream.avail_in = MAGIC_NUMBER;

			read++;
			bytescompressed += MAGIC_NUMBER;
		}
		else if (bytescompressed == length)
		{
			stream.avail_in = 0;
			stream.next_in = input_buffer;

			read ++;
		}
		else
		{
			int size = length - read*MAGIC_NUMBER;
			memcpy (input_buffer, src + size, MAGIC_NUMBER);
			bytescompressed += size;
			stream.avail_in = size;

			read ++;
		}


		stream.next_in = input_buffer;
	}

	err = deflate (&stream, Z_NO_FLUSH);

	unsigned long count = output_length - stream.avail_out;
	if (count)
	{
		memcpy ((unsigned char*)dst + decompressed, output_buffer, count);
		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;

		memset (output_buffer, 0, MAGIC_NUMBER);
	}

	if (err != Z_OK)
		break;
}

for (; ;)
{
	err = deflate (&stream, Z_FINISH);
	
	unsigned int count = output_length - stream.avail_out;
	if (count)
	{
		memcpy ((unsigned char*)dst + decompressed, output_buffer, count);
		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;

		memset (input_buffer, 0, MAGIC_NUMBER);
	}

	if (err != Z_OK)
		break;
}

deflateEnd (&stream);
if ((err != Z_OK) && (err != Z_STREAM_END))
{
	printf ("zlib:%i, %s\n", err, stream.msg);
	inflateEnd (&stream);
	return err;
}

fwrite (dst, 1, decompressed, fout);

fclose (fin);
fclose (fout);

For my test, I used a random pdf file to compress/decompress. Using the file method, the original file is 36393, and compressed its 29750. Working on the file in memory, the compressed version comes out as 28521. The last if functions 'err' returns 1, which means that Z_STREAM_END, but its not producing the same results as the other method? Any ideas, suggestions? Thanks for the help! -brad

Share this post


Link to post
Share on other sites
Galapaegos    277
I'm using chunks of 4096 bytes to send to the zlib api first. Both functions return the same bytes back out. I've followed both loops until they reach the point when they differ, and it happens in the 2nd for loop. The second to last chunk is where the functions receive different output from the deflate function. Both functions receive the same amount of input, and I'd assume they would receive the same output, no?

-brad

Share this post


Link to post
Share on other sites
Galapaegos    277
Also, I thought I should mention return values. I have been watching the return values and the z_stream.msg attribute closely. After I have sent all the input to the deflate function, I recieve the -5 signal telling me to switch the flush from Z_NO_FLUSH to Z_FINISH, and the msg is "buffer error". Both functions do this, so it doesn't sound like my code is messing up?

I receive the return code '1' for both functions, even though the one ends prematurely. The msg attribute doesn't update, so I assume everything went well? Any thoughts/ideas?

-brad

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this