Jump to content
  • Advertisement
Sign in to follow this  
Galapaegos

zlib issue

This topic is 4354 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi Folks, Our file format uses zlib to compress/decompress files. Essentially, I'm trying to write a new compress/decompress routine to add my own progress meter. I have two test functions, one that loads a file and compresses it reading via 'fread/fwrite' calls, and another function that compresses by using memcpy. The memory mapping function doesn't return the same size as the other method. I need to get the memory mapping function to work, since this is the situation that is happening in my loading function. Here are the two functions for reference: file:
FILE *fin = fopen (arg.c_str (), "rb");
FILE *fout = fopen (out.c_str (), "wb");

//z_stream
z_stream stream;
memset (&stream, 0, sizeof (stream));

int err = deflateInit (&stream, 6);
if (err != Z_OK)
	printf ("zlib:%s\n", stream.msg);

int input_length = MAGIC_NUMBER;
unsigned char input_buffer [MAGIC_NUMBER];
int output_length = MAGIC_NUMBER;
unsigned char output_buffer [MAGIC_NUMBER];

stream.next_in = input_buffer;
stream.next_out = output_buffer;

stream.avail_in = fread (input_buffer, 1, input_length, fin);
stream.avail_out = MAGIC_NUMBER;

int read = 1, bytescompressed = MAGIC_NUMBER, decompressed = 0;

for (; ;)
{
	if (stream.avail_in == 0)
	{
		stream.avail_in = fread (input_buffer, 1, input_length, fin);
		stream.next_in = input_buffer;

		read ++;
	}

	err = deflate (&stream, Z_NO_FLUSH);

	unsigned long count = output_length - stream.avail_out;
	if (count)
	{
		if (fwrite (output_buffer, 1, count, fout) != count)
		{
			err = Z_ERRNO;
			return err;
		}

		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;
	}

	if (err != Z_OK)
		break;
}

for (; ;)
{
	err = deflate (&stream, Z_FINISH);
	
	unsigned int count = output_length - stream.avail_out;
	if (count)
	{
		if (fwrite (output_buffer, 1, count, fout) != count)
		{
			err = Z_ERRNO;
			return err;
		}

		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;
	}

	if (err != Z_OK)
		break;
}

deflateEnd (&stream);
if ((err != Z_OK) && (err != Z_STREAM_END))
{
	printf ("zlib:%i, %s\n", err, stream.msg);
	inflateEnd (&stream);
	return err;
}

printf ("decompressed:%i\n", decompressed);

fclose (fin);
fclose (fout);

memory mapped:
FILE *fin = fopen (arg.c_str (), "rb");
FILE *fout = fopen (out.c_str (), "wb");

int length = filelength (fileno (fin));
int destLength = length*1.001 + 12;

unsigned char *src = new unsigned char [length];
unsigned char *dst = new unsigned char [destLength];

fread (src, 1, length, fin);

//z_stream
z_stream stream;
memset (&stream, 0, sizeof (stream));

int err = deflateInit (&stream, 6);
if (err != Z_OK)
	printf ("zlib:%s\n", stream.msg);

int input_length = MAGIC_NUMBER;
unsigned char input_buffer [MAGIC_NUMBER];
int output_length = MAGIC_NUMBER;
unsigned char output_buffer [MAGIC_NUMBER];

memset (input_buffer, 0, MAGIC_NUMBER);
memset (output_buffer, 0, MAGIC_NUMBER);

stream.next_in = input_buffer;
stream.next_out = output_buffer;

memcpy (input_buffer, src, MAGIC_NUMBER);
stream.avail_in = MAGIC_NUMBER;
stream.avail_out = MAGIC_NUMBER;

int read = 1, bytescompressed = MAGIC_NUMBER, decompressed = 0;

for (; ;)
{
	if (stream.avail_in == 0)
	{
		memset (input_buffer, 0, MAGIC_NUMBER);

		if ((read+1)*MAGIC_NUMBER < length)
		{
			memcpy (input_buffer, src + read*MAGIC_NUMBER, MAGIC_NUMBER);
			stream.avail_in = MAGIC_NUMBER;

			read++;
			bytescompressed += MAGIC_NUMBER;
		}
		else if (bytescompressed == length)
		{
			stream.avail_in = 0;
			stream.next_in = input_buffer;

			read ++;
		}
		else
		{
			int size = length - read*MAGIC_NUMBER;
			memcpy (input_buffer, src + size, MAGIC_NUMBER);
			bytescompressed += size;
			stream.avail_in = size;

			read ++;
		}


		stream.next_in = input_buffer;
	}

	err = deflate (&stream, Z_NO_FLUSH);

	unsigned long count = output_length - stream.avail_out;
	if (count)
	{
		memcpy ((unsigned char*)dst + decompressed, output_buffer, count);
		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;

		memset (output_buffer, 0, MAGIC_NUMBER);
	}

	if (err != Z_OK)
		break;
}

for (; ;)
{
	err = deflate (&stream, Z_FINISH);
	
	unsigned int count = output_length - stream.avail_out;
	if (count)
	{
		memcpy ((unsigned char*)dst + decompressed, output_buffer, count);
		decompressed += count;

		stream.next_out = output_buffer;
		stream.avail_out = output_length;

		memset (input_buffer, 0, MAGIC_NUMBER);
	}

	if (err != Z_OK)
		break;
}

deflateEnd (&stream);
if ((err != Z_OK) && (err != Z_STREAM_END))
{
	printf ("zlib:%i, %s\n", err, stream.msg);
	inflateEnd (&stream);
	return err;
}

fwrite (dst, 1, decompressed, fout);

fclose (fin);
fclose (fout);

For my test, I used a random pdf file to compress/decompress. Using the file method, the original file is 36393, and compressed its 29750. Working on the file in memory, the compressed version comes out as 28521. The last if functions 'err' returns 1, which means that Z_STREAM_END, but its not producing the same results as the other method? Any ideas, suggestions? Thanks for the help! -brad

Share this post


Link to post
Share on other sites
Advertisement
I'm using chunks of 4096 bytes to send to the zlib api first. Both functions return the same bytes back out. I've followed both loops until they reach the point when they differ, and it happens in the 2nd for loop. The second to last chunk is where the functions receive different output from the deflate function. Both functions receive the same amount of input, and I'd assume they would receive the same output, no?

-brad

Share this post


Link to post
Share on other sites
Also, I thought I should mention return values. I have been watching the return values and the z_stream.msg attribute closely. After I have sent all the input to the deflate function, I recieve the -5 signal telling me to switch the flush from Z_NO_FLUSH to Z_FINISH, and the msg is "buffer error". Both functions do this, so it doesn't sound like my code is messing up?

I receive the return code '1' for both functions, even though the one ends prematurely. The msg attribute doesn't update, so I assume everything went well? Any thoughts/ideas?

-brad

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!