Sign in to follow this  
Afr0m@n

[.net] Reading binary files problem (DBPF format)

Recommended Posts

Ok, so for the last couple days I've been working on a dll to load Maxis' DBPF 1 format for TSO (The Sims Online). Things are progressing slowly, but I'm not quite there yet. And I can't figure out what I'm doing wrong either. :\ Currently, when running these functions from my test application, I get a ton of files (specifically, with the archive I'm testing on - 1039) that are each 13 kbytes. They don't have a fileextension, but luckily at least they have a filename ^^. If someone could help me figure out what I'm doing wrong, I'd be extremely happy! :) Here's my code so far: DBPF1File.cs:
using System;
using System.Collections.Generic;
using System.Text;

namespace DBPF
{
    class DBPF1File
    {
        public UInt32 m_TypeID;
        public UInt32 m_GroupID;
        public UInt32 m_InstanceID;
        public UInt32 m_Offset;
        public UInt32 m_Size;
        public byte[] m_Data;
    }
}


DBPF1Loader.cs:
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Windows.Forms;

namespace DBPF
{
    public class DBPF1Loader
    {
        //General data about the file loaded.
        private string m_Header;
        private UInt32 m_Version;
        private UInt32 m_ArchiveID;
        private UInt32 m_ArchiveType;
        private UInt32 m_NumberOfFiles;
        private UInt32 m_DirectoryOffset;
        private UInt32 m_DirectorySize;

        private DBPF1File[] m_FileList;

        public DBPF1Loader(string Filename)
        {
            LoadArchive(Filename);
        }

        private void LoadArchive(string Filename)
        {
            FileStream Stream = File.OpenRead(Filename);
            BinaryReader Reader = new BinaryReader(Stream);
            byte[] Data = new byte[4];

            //Read the header (4 bytes), and convert them from
            //a binary to an ASCII encoding.
            ASCIIEncoding Encoding = new ASCIIEncoding();
            m_Header = Encoding.GetString(Reader.ReadBytes(4));

            //Read the version signature.
            m_Version = Reader.ReadUInt32();

            //Read 16 null bytes from file, no idea what these are for.
            for (int i = 0; i <= 15; i++)
                Reader.ReadByte();

            //Read the archive id.
            m_ArchiveID = Reader.ReadUInt32();

            //Read the archive type.
            m_ArchiveType = Reader.ReadUInt32();

            //Read an unknown UInt32.
            Reader.ReadUInt32();

            //Read the total number of files in this archive.
            m_NumberOfFiles = Reader.ReadUInt32();
            
            //Read the (initial) directory offset in this archive.
            m_DirectoryOffset = Reader.ReadUInt32();

            //Read the directory size in this archive.
            m_DirectorySize = Reader.ReadUInt32();

            //Read 48 unknown bytes.
            for (int i = 0; i <= 47; i++)
                Reader.ReadByte();

            //Find the directory offset in the stream
            Stream.Seek((long)m_DirectoryOffset, SeekOrigin.Current);

            //The current offset in the stream at this point.
            long CurrentOffset = Stream.Position;

            //Array of files loaded by the loader.
            //Initialized here because the total number of files is now known.
            m_FileList = new DBPF1File[m_NumberOfFiles];

            //Cycle through the number of files in the archive.
            for (int y = 0; y < m_NumberOfFiles; y++)
            {
                DBPF1File TmpFile = new DBPF1File();

                TmpFile.m_TypeID = Reader.ReadUInt32();
                TmpFile.m_GroupID = Reader.ReadUInt32();
                TmpFile.m_InstanceID = Reader.ReadUInt32();
                TmpFile.m_Offset = Reader.ReadUInt32();
                TmpFile.m_Size = Reader.ReadUInt32();
                TmpFile.m_Data = new byte[TmpFile.m_Size];

                //Seek to offset of the filedata,
                Stream.Seek(TmpFile.m_Offset, SeekOrigin.Current);
                //read the number of bytes specified in m_Size,
                TmpFile.m_Data = Reader.ReadBytes((int)TmpFile.m_Size);
                //and seek back again to the starting point
                //from the beginning of the stream.
                Stream.Seek(CurrentOffset, SeekOrigin.Begin);

                //For each file, save the position in the stream
                //at this point.
                CurrentOffset = Stream.Position;

                //Add the temporary file to the list of files loaded by
                //the loader.
                m_FileList[y] = TmpFile;
            }
        }

        public void UnpackArchive(string Filename)
        {
            //Filetype extension for the current file.
            string CurrentExtension = "";
            //Incrementor for file names, because the actual
            //filenames aren't known.
            int CurrentFileName = 0;

            Directory.CreateDirectory(Filename);

            foreach (DBPF1File TmpFile in m_FileList)
            {
                /*1 - BMP image - (compressed)
                2 - TGA image - (compressed)
                5 - SKEL skeleton - (compressed)
                7 - ANIM animation - (compressed)
                9 - MESH model - (compressed)
                11 - BND binding - (uncompressed)
                12 - APR appearance - (uncompressed)
                13 - OTF outfit - (uncompressed)
                14 - PNG image - (uncompressed)
                15 - PO purchasable object (uncompressed)
                16 - COL collection - (compressed)
                18 - HAG group - (compressed)
                19 - JPEG image - (compressed)
                20 - JPEG image - (uncompressed)*/

                CurrentFileName++;

                switch (TmpFile.m_TypeID)
                {
                    case 1:
                        CurrentExtension = ".bmp";
                        break;
                    case 2:
                        CurrentExtension = ".tga";
                        break;
                    case 5:
                        CurrentExtension = ".skel";
                        break;
                    case 7:
                        CurrentExtension = ".anim";
                        break;
                    case 9:
                        CurrentExtension = ".mesh";
                        break;
                    case 11:
                        CurrentExtension = ".bnd";
                        break;
                    case 12:
                        CurrentExtension = ".apr";
                        break;
                    case 13:
                        CurrentExtension = ".otf";
                        break;
                    case 14:
                        CurrentExtension = ".png";
                        break;
                    case 15:
                        CurrentExtension = ".po";
                        break;
                    case 16:
                        CurrentExtension = ".col";
                        break;
                    case 18:
                        CurrentExtension = ".hag";
                        break;
                    case 19:
                        CurrentExtension = ".jpg";
                        break;
                    case 20:
                        CurrentExtension = ".jpg";
                        break;
                }

                FileStream Stream = File.Create(Filename + "\\" + CurrentFileName.ToString() + CurrentExtension);
                BinaryWriter Writer = new BinaryWriter(Stream);

                Writer.Write(TmpFile.m_Data);

                Writer.Close();
                Stream.Close();
            }
        }

        public string Header
        {
            get{return m_Header;}
        }

        public UInt32 Version
        {
            get { return m_Version; }
        }

        public UInt32 ArchiveType
        {
            get { return m_ArchiveType; }
        }

        public UInt32 ArchiveID
        {
            get { return m_ArchiveID; }
        }

        public UInt32 NumberOfFiles
        {
            get { return m_NumberOfFiles; }
        }

        public UInt32 DirectoryOffset
        {
            get { return m_DirectoryOffset; }
        }

        public UInt32 DirectorySize
        {
            get { return m_DirectorySize; }
        }
    }
}


Here's links to the file format specifications for DBPF 1 and 2: DBPF1 DBPF2 Thanks in advance for any help! :) Edit: And YES, I am sure that the archive I'm currently testing this out on is a DBPF1 archive, because the header says 'DBPF', not 'FAR!byAZ'.

Share this post


Link to post
Share on other sites
You don't have a default case in your switch statement; make it use an extension like ".xyz" in the default case so you know it has an ID that's out of bounds.

Also, this doesn't look right:


Stream.Seek(CurrentOffset, SeekOrigin.Begin);
// comment
CurrentOffset = Stream.Position;


It looks like this is your program flow when reading the files:

*save offset

for each tempfile:
* read header
* set offset to data and read data
* restore offset
* save offset

If I'm not making a mistake, offset will be the same thing every time, so you'll read the first file m_NumberOfFiles times. Move the line 'CurrentOffset = Stream.Position' four lines earlier, so you save the position right before you first seek to the start of the data.

Also, you're loading all the data into memory then saving it all to disk. That's not good for huge archive files. (I don't know if this game includes one big enough to cause problems, though.) Instead, you could only read in the headers in your first pass, and not read the data until you're making the output files. This would save memory and also make your code simpler: the first time through, you could read in all the headers without messing with the file offset, and when you're writing the data, you could jump to the offsets in your DBPF1File entries without worrying about saving and restoring the offset.

Share this post


Link to post
Share on other sites
Quote:
Original post by nagromo
You don't have a default case in your switch statement; make it use an extension like ".xyz" in the default case so you know it has an ID that's out of bounds.

Also, this doesn't look right:


Stream.Seek(CurrentOffset, SeekOrigin.Begin);
// comment
CurrentOffset = Stream.Position;


It looks like this is your program flow when reading the files:

*save offset

for each tempfile:
* read header
* set offset to data and read data
* restore offset
* save offset

If I'm not making a mistake, offset will be the same thing every time, so you'll read the first file m_NumberOfFiles times. Move the line 'CurrentOffset = Stream.Position' four lines earlier, so you save the position right before you first seek to the start of the data.


Awesome, that worked! :) Thanks a bunch! :) Now I just have to figure out why the IDs of the files are out of bounds - the loading of the files seems to work perfectly. :) I'm going to keep messing around a bit with the code to see if reading different offsets from different places in file could have anything to say for the file IDs.

Also, in regards to saving memory when loading files - This is very useful, thanks again. Once I have time, I might try to redesign the loading, but right now, actually making the loading work is my highest priority. Thing is though, I would like to have some way of viewing the extracted files directly in the program using my dll when the files have been unpacked. Maybe I should just store the complete paths for all the files in the archive once they have been extracted and load them manually 1 by 1 when the user wants to see them. I don't really know yet. :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this