Jump to content
  • Advertisement
Sign in to follow this  
JohnnyCasil

Random zlib problem [Solved]

This topic is 4577 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I am currently adding package file support to the filesystem of my engine. I am using zlib to do this for it's ease of use, but unfortunately I have run into a rather strange problem. I created a rather simple pack file format, nothing but a header that tells me how many files I have packed, a file table and then the compressed data. The problem is when I use zlib to uncompress the data I get wierd results with small text files. An example input file would be
Come on zlib, work!


Which when uncompressed would give me the output of
Come on zlib, woÍÍÍ


Now if I take a large amount of new lines at the end of the file, or make the file larger, everything works out fine. Everything also works out fine with binary files. Is there some piece of documentation that I missed dealing with small text files, or is this a result of the compression algorithm? Any insight into this would be helpful, thank you. [Edited by - JohnnyCasil on January 2, 2006 8:26:52 AM]

Share this post


Link to post
Share on other sites
Advertisement
Looks like a bug in your code, either a buffer that is too small, or not properly cleared. That's about what I can venture to say, I've made my own little simple filesystem like you have with ZLIB and it works great, so I doubt there's anything wrong with your zlib version. If you want to see my code, here's the project. Maybe you can show us some of the revelant code, I'm not sure where it'd be though, but you can look into the code that compresses it first.

Share this post


Link to post
Share on other sites
Quote:
Original post by JohnnyCasil
Now if I take a large amount of new lines at the end of the file, or make the file larger, everything works out fine. Everything also works out fine with binary files. Is there some piece of documentation that I missed dealing with small text files, or is this a result of the compression algorithm? Any insight into this would be helpful, thank you.


Are you using the streaming API, the buffer API, ot the file API of zlib?

Share this post


Link to post
Share on other sites
Well, I am now completely perplexed as to why this doesn't work. I don't know what I changed, or what happened, but now when I uncompress text data, all I get are "ÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ" no matter what I input. But binary files work perfectly fine and uncompress fine as well. The way I have it set up is a wrote a Java application for a quick gui file packer. All of the gui stuff is insignificant, here is the Java file packer class. Mind you, the code is messy but this is because I just wanted a quick prototype to see if I could get this to work.

package packer;

import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.ArrayList;
import java.util.zip.Deflater;

public class FilePacker
{
private DataOutputStream m_DOS;
private ArrayList<File> m_FileList;

private class FILE_ENTRY
{
public byte name [];
public int size;
public int compress_size;
public int offset;

public FILE_ENTRY ()
{
name = new byte [64];
size = 0;
compress_size = 0;
offset = 0;
}
}

private FILE_ENTRY[] m_FileTable;

public FilePacker () throws FileNotFoundException
{
m_DOS = new DataOutputStream(new FileOutputStream("out.dpk"));
m_FileList = new ArrayList<File>();
}

public void pack () throws IOException
{
writeHeader();
createFileTable();
writeFileTable();
writeCompressedFiles();
m_DOS.close();
}

public void addFile (File file)
{
m_FileList.add(file);
}

private void writeHeader () throws IOException
{
ByteBuffer bb;

m_DOS.writeBytes(".dpk"); // Header
m_DOS.writeByte(1); // Version number
m_DOS.writeByte(1); // Compression (0 - no / 1 -yes)
m_DOS.writeByte(0); // Encryption (0 - no / 1 - yes)
m_DOS.writeByte(0); // Reserved / Padding
}

private void createFileTable () throws IOException
{
m_FileTable = new FILE_ENTRY [m_FileList.size()];

int offset = 12 + ((64 + 12) * m_FileList.size());

ByteBuffer bb = ByteBuffer.allocate(4);
bb.order(ByteOrder.nativeOrder());
bb.putInt(m_FileList.size());
byte s [] = bb.array();
m_DOS.write(s);

for (int t = 0; t < m_FileList.size(); t++)
{
m_FileTable[t] = new FILE_ENTRY();

File f = m_FileList.get(t);
byte[] name = f.getName().getBytes();
copyBytes(m_FileTable[t].name, name);
m_FileTable[t].size = (int)f.length();
m_FileTable[t].compress_size = compressedSize(f);
m_FileTable[t].offset = offset;

System.out.println(offset);
offset += m_FileTable[t].compress_size;
}
}

private void writeFileTable () throws IOException
{
for (int t = 0; t < m_FileTable.length; t++)
{
m_DOS.write(m_FileTable[t].name);
ByteBuffer bb = ByteBuffer.allocate(12);
bb.order(ByteOrder.nativeOrder());

bb.putInt(m_FileTable[t].size);
bb.putInt(m_FileTable[t].compress_size);
bb.putInt(m_FileTable[t].offset);

byte data [] = bb.array();
m_DOS.write(data);
}
}

private void copyBytes (byte[] b1, byte[] b2)
{
for (int t = 0; t < b1.length; t++)
{
if (t >= b2.length)
break;

b1[t] = b2[t];
}
}

private int compressedSize (File f) throws IOException
{
FileInputStream fis = new FileInputStream(f);
byte data [] = new byte[(int)f.length()];
byte output [] = new byte[(int)f.length()];
fis.read(data);
fis.close();
Deflater compressor = new Deflater();
compressor.setInput(data);
compressor.finish();
int size = compressor.deflate(output);
data = null;
output = null;
System.gc();
return size;
}

private void writeCompressedFiles () throws IOException
{
for (int t = 0; t < m_FileList.size(); t++)
{
File f = m_FileList.get(t);

FileInputStream fis = new FileInputStream(f);
byte data [] = new byte[(int)f.length()];
byte output [] = new byte[(int)f.length()];
fis.read(data);
Deflater compressor = new Deflater();
compressor.setInput(data);
compressor.finish();
int size = compressor.deflate(output);

ByteBuffer bb = ByteBuffer.wrap(output, 0, size);
bb.order(ByteOrder.nativeOrder());
byte fin [] = bb.array();


m_DOS.write(output);
//m_DOS.write(data, 0, (int)f.length());
}
}
}



And here is a quick unpacking utility I wrote in C++ to test to see if the package files are correct.


#pragma comment(lib, "zlib.lib")

#include <zlib.h> // Include zlib
#include <iostream>
using namespace std;

#pragma pack(1)
struct PACKAGE_FILE_HEADER
{
char header [4];
unsigned char version;
unsigned char compression;
unsigned char encryption;
unsigned char buffer;
};

struct PACKAGE_FILE_ENTRY
{
char name [64];
int size;
int compressed_size;
int offset;
};

void main ()
{
FILE* pack = fopen("out.dpk", "rb");

PACKAGE_FILE_HEADER pfh;
fread(&pfh, sizeof(PACKAGE_FILE_HEADER), 1, pack);

cout << "Package File Header" << endl;
cout << "-- Header: " << pfh.header[0] << pfh.header[1] << pfh.header[2] << pfh.header[3] <<endl;
cout << "-- Version: " << (int)pfh.version << endl;
cout << "-- Compression: " << (int)pfh.compression << endl;
cout << "-- Encryption: " << (int)pfh.encryption << endl;
cout << "-- Padding: " << (int)pfh.buffer << endl;

int numberFiles = 0;
fread(&numberFiles, sizeof(int), 1, pack);
cout << "Package file contains " << (int)numberFiles << " files" << endl;

PACKAGE_FILE_ENTRY* entry = new PACKAGE_FILE_ENTRY[numberFiles];
fread(entry, sizeof(PACKAGE_FILE_ENTRY), numberFiles, pack);

cout << "Package File Entries" << endl;
for (int t = 0; t < numberFiles; t++)
{
cout << "-- File Entry " << t << endl;
cout << "---- Name: " << entry[t].name << endl;
cout << "---- Size: " << entry[t].size << endl;
cout << "---- Compressed Size: " << entry[t].compressed_size << endl;
cout << "---- Offset: " << entry[t].offset << endl;
}

for (int t = 0; t < numberFiles; t++)
{
FILE* output = fopen(entry[t].name, "wb");
unsigned char* compressed = new unsigned char[entry[t].compressed_size];
unsigned char* data = new unsigned char[entry[t].size];

fseek(pack, entry[t].offset, SEEK_SET);
fread(compressed, sizeof(unsigned char), entry[t].compressed_size, pack);

unsigned long size = entry[t].size;

uncompress(data, &size, compressed, entry[t].compressed_size);

fwrite(data, 1, entry[t].size, output);

delete [] compressed;

delete [] data;
fclose(output);
}

system("pause");
}



My header, the file table, and the number of files all come out fine. Binary files uncompress fine. I don't understand what is wrong here, everything seems fine to me. I am sure it is probably just some stupid thing I am overlooking, but it is so frustrating.

Share this post


Link to post
Share on other sites
Quote:
Original post by JohnnyCasil
And here is a quick unpacking utility I wrote in C++ to test to see if the package files are correct.


Ah, you're using thye buffer-level API. I'm less familiar with it, I usually use the streaming API since I don't know the original size before decompression.

Here's a suggestion. In the streaming-level API it's oftent necessary to check the return status of the decompress call to see if decompression was complete, and make a final "fulsh buffer" call to get the last dregs out. The symptom of not doing that is exactly what you're experiencing. Check the API docs to see if that's necessary with the API you're using.

Share this post


Link to post
Share on other sites
Hmm, valuable lesson, always check your return values. For the text methods the uncompress is infact returning an Z_DATA_ERROR return, which it would return if the input data is corrupted. Now I don't understand why I would be getting a data error ONLY for the text data.

Share this post


Link to post
Share on other sites
One thing that may be happening is that when you have lines, such as: byte data [] = new byte[(int)f.length()]; and you use text, it really should be null terminated, so that should be: byte data [] = new byte[(int)f.length() + 1];, then data[f.length() + 1] = '\0'; or however you do it in Java. I'm not that familure with Java, so I'm not sure if that will work as is. So definitly give that a try out, and don't forget to adjust all your sizes now when you add that extra byte.

Share this post


Link to post
Share on other sites
I gave that a shot and I still get the same errors. I don't think that is the problem because I am not treating the text any differently from raw data, so as far as zlib can see I am just sending it arbitrary data. That is why I am so confused, because regardless if the file is binary or text, I load it as binary so I can just get the raw bytes, but something is going wrong with files that are text. What I can't seem to figure out. I just got rid of the compression to test and make sure everything is working as far as the file IO goes, and without compression everything turns out fine.

Share this post


Link to post
Share on other sites
Well, I solved the problem. Apparently despite the fact that Java's documentation makes no mention of it, when you use the Java delfate class to compress things you STILL need to provide the compresser with a buffer that its (size of input) * 1.1 + 12. I was assuming that Java was compensating for this behind the scenes since it failed to tell me. And this now explains why binary files worked (They were compressing so the output compresser would need a smaller buffer) and the text files didn't (They didn't compress so zlib bloated them). Thank you guys for the help though anyways, without it I may not have stumbled upon this because I was originally looking the completely wrong direction.

Share this post


Link to post
Share on other sites
Quote:
Original post by JohnnyCasil
Apparently despite the fact that Java's documentation makes no mention of it, when you use the Java delfate class to compress things you STILL need to provide the compresser with a buffer that its (size of input) * 1.1 + 12. I was assuming that Java was compensating for this behind the scenes since it failed to tell me.


Tricky tricky! I'll have to keep that in mind for future problems. Glad you figured that out!

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!