Back to General and Gameplay Programming

[.net] Reading binary files problem (DBPF format)

General and Gameplay Programming Programming

Started by Afr0m@n March 10, 2007 05:07 AM

2 comments, last by Afr0m@n 17 years, 1 month ago

Afr0m@n

100

Author

March 10, 2007 05:07 AM

Ok, so for the last couple days I've been working on a dll to load Maxis' DBPF 1 format for TSO (The Sims Online). Things are progressing slowly, but I'm not quite there yet. And I can't figure out what I'm doing wrong either. :\ Currently, when running these functions from my test application, I get a ton of files (specifically, with the archive I'm testing on - 1039) that are each 13 kbytes. They don't have a fileextension, but luckily at least they have a filename ^^. If someone could help me figure out what I'm doing wrong, I'd be extremely happy! :) Here's my code so far: DBPF1File.cs:

using System;
using System.Collections.Generic;
using System.Text;

namespace DBPF
{
    class DBPF1File
    {
        public UInt32 m_TypeID;
        public UInt32 m_GroupID;
        public UInt32 m_InstanceID;
        public UInt32 m_Offset;
        public UInt32 m_Size;
        public byte[] m_Data;
    }
}

DBPF1Loader.cs:


using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Windows.Forms;

namespace DBPF
{
    public class DBPF1Loader
    {
        //General data about the file loaded.
        private string m_Header;
        private UInt32 m_Version;
        private UInt32 m_ArchiveID;
        private UInt32 m_ArchiveType;
        private UInt32 m_NumberOfFiles;
        private UInt32 m_DirectoryOffset;
        private UInt32 m_DirectorySize;

        private DBPF1File[] m_FileList;

        public DBPF1Loader(string Filename)
        {
            LoadArchive(Filename);
        }

        private void LoadArchive(string Filename)
        {
            FileStream Stream = File.OpenRead(Filename);
            BinaryReader Reader = new BinaryReader(Stream);
            byte[] Data = new byte[4];

            //Read the header (4 bytes), and convert them from
            //a binary to an ASCII encoding.
            ASCIIEncoding Encoding = new ASCIIEncoding();
            m_Header = Encoding.GetString(Reader.ReadBytes(4));

            //Read the version signature.
            m_Version = Reader.ReadUInt32();

            //Read 16 null bytes from file, no idea what these are for.
            for (int i = 0; i <= 15; i++)
                Reader.ReadByte();

            //Read the archive id.
            m_ArchiveID = Reader.ReadUInt32();

            //Read the archive type.
            m_ArchiveType = Reader.ReadUInt32();

            //Read an unknown UInt32.
            Reader.ReadUInt32();

            //Read the total number of files in this archive.
            m_NumberOfFiles = Reader.ReadUInt32();
            
            //Read the (initial) directory offset in this archive.
            m_DirectoryOffset = Reader.ReadUInt32();

            //Read the directory size in this archive.
            m_DirectorySize = Reader.ReadUInt32();

            //Read 48 unknown bytes.
            for (int i = 0; i <= 47; i++)
                Reader.ReadByte();

            //Find the directory offset in the stream
            Stream.Seek((long)m_DirectoryOffset, SeekOrigin.Current);

            //The current offset in the stream at this point.
            long CurrentOffset = Stream.Position;

            //Array of files loaded by the loader.
            //Initialized here because the total number of files is now known.
            m_FileList = new DBPF1File[m_NumberOfFiles];

            //Cycle through the number of files in the archive.
            for (int y = 0; y < m_NumberOfFiles; y++)
            {
                DBPF1File TmpFile = new DBPF1File();

                TmpFile.m_TypeID = Reader.ReadUInt32();
                TmpFile.m_GroupID = Reader.ReadUInt32();
                TmpFile.m_InstanceID = Reader.ReadUInt32();
                TmpFile.m_Offset = Reader.ReadUInt32();
                TmpFile.m_Size = Reader.ReadUInt32();
                TmpFile.m_Data = new byte[TmpFile.m_Size];

                //Seek to offset of the filedata,
                Stream.Seek(TmpFile.m_Offset, SeekOrigin.Current);
                //read the number of bytes specified in m_Size,
                TmpFile.m_Data = Reader.ReadBytes((int)TmpFile.m_Size);
                //and seek back again to the starting point
                //from the beginning of the stream.
                Stream.Seek(CurrentOffset, SeekOrigin.Begin);

                //For each file, save the position in the stream
                //at this point.
                CurrentOffset = Stream.Position;

                //Add the temporary file to the list of files loaded by
                //the loader.
                m_FileList[y] = TmpFile;
            }
        }

        public void UnpackArchive(string Filename)
        {
            //Filetype extension for the current file.
            string CurrentExtension = "";
            //Incrementor for file names, because the actual
            //filenames aren't known.
            int CurrentFileName = 0;

            Directory.CreateDirectory(Filename);

            foreach (DBPF1File TmpFile in m_FileList)
            {
                /*1 - BMP image - (compressed)
                2 - TGA image - (compressed)
                5 - SKEL skeleton - (compressed)
                7 - ANIM animation - (compressed)
                9 - MESH model - (compressed)
                11 - BND binding - (uncompressed)
                12 - APR appearance - (uncompressed)
                13 - OTF outfit - (uncompressed)
                14 - PNG image - (uncompressed)
                15 - PO purchasable object (uncompressed)
                16 - COL collection - (compressed)
                18 - HAG group - (compressed)
                19 - JPEG image - (compressed)
                20 - JPEG image - (uncompressed)*/

                CurrentFileName++;

                switch (TmpFile.m_TypeID)
                {
                    case 1:
                        CurrentExtension = ".bmp";
                        break;
                    case 2:
                        CurrentExtension = ".tga";
                        break;
                    case 5:
                        CurrentExtension = ".skel";
                        break;
                    case 7:
                        CurrentExtension = ".anim";
                        break;
                    case 9:
                        CurrentExtension = ".mesh";
                        break;
                    case 11:
                        CurrentExtension = ".bnd";
                        break;
                    case 12:
                        CurrentExtension = ".apr";
                        break;
                    case 13:
                        CurrentExtension = ".otf";
                        break;
                    case 14:
                        CurrentExtension = ".png";
                        break;
                    case 15:
                        CurrentExtension = ".po";
                        break;
                    case 16:
                        CurrentExtension = ".col";
                        break;
                    case 18:
                        CurrentExtension = ".hag";
                        break;
                    case 19:
                        CurrentExtension = ".jpg";
                        break;
                    case 20:
                        CurrentExtension = ".jpg";
                        break;
                }

                FileStream Stream = File.Create(Filename + "\\" + CurrentFileName.ToString() + CurrentExtension);
                BinaryWriter Writer = new BinaryWriter(Stream);

                Writer.Write(TmpFile.m_Data);

                Writer.Close();
                Stream.Close();
            }
        }

        public string Header
        {
            get{return m_Header;}
        }

        public UInt32 Version
        {
            get { return m_Version; }
        }

        public UInt32 ArchiveType
        {
            get { return m_ArchiveType; }
        }

        public UInt32 ArchiveID
        {
            get { return m_ArchiveID; }
        }

        public UInt32 NumberOfFiles
        {
            get { return m_NumberOfFiles; }
        }

        public UInt32 DirectoryOffset
        {
            get { return m_DirectoryOffset; }
        }

        public UInt32 DirectorySize
        {
            get { return m_DirectorySize; }
        }
    }
}

Here's links to the file format specifications for DBPF 1 and 2: DBPF1 DBPF2 Thanks in advance for any help! :) Edit: And YES, I am sure that the archive I'm currently testing this out on is a DBPF1 archive, because the header says 'DBPF', not 'FAR!byAZ'.

_______________________Afr0Games

Afr0m@n

100

Author

March 12, 2007 03:05 AM

Anyone? :(

I really can't figure this out on my own. :\

_______________________Afr0Games

nagromo

676

March 12, 2007 10:28 PM

You don't have a default case in your switch statement; make it use an extension like ".xyz" in the default case so you know it has an ID that's out of bounds.

Also, this doesn't look right:

Stream.Seek(CurrentOffset, SeekOrigin.Begin);// commentCurrentOffset = Stream.Position;

It looks like this is your program flow when reading the files:

*save offset

for each tempfile:
* read header
* set offset to data and read data
* restore offset
* save offset

If I'm not making a mistake, offset will be the same thing every time, so you'll read the first file m_NumberOfFiles times. Move the line 'CurrentOffset = Stream.Position' four lines earlier, so you save the position right before you first seek to the start of the data.

Also, you're loading all the data into memory then saving it all to disk. That's not good for huge archive files. (I don't know if this game includes one big enough to cause problems, though.) Instead, you could only read in the headers in your first pass, and not read the data until you're making the output files. This would save memory and also make your code simpler: the first time through, you could read in all the headers without messing with the file offset, and when you're writing the data, you could jump to the offsets in your DBPF1File entries without worrying about saving and restoring the offset.

Afr0m@n

100

Author

March 13, 2007 01:02 PM

Quote:Original post by nagromo
You don't have a default case in your switch statement; make it use an extension like ".xyz" in the default case so you know it has an ID that's out of bounds.

Also, this doesn't look right:
Stream.Seek(CurrentOffset, SeekOrigin.Begin);// commentCurrentOffset = Stream.Position;
It looks like this is your program flow when reading the files:

*save offset

for each tempfile:
* read header
* set offset to data and read data
* restore offset
* save offset

If I'm not making a mistake, offset will be the same thing every time, so you'll read the first file m_NumberOfFiles times. Move the line 'CurrentOffset = Stream.Position' four lines earlier, so you save the position right before you first seek to the start of the data.

Awesome, that worked! :) Thanks a bunch! :) Now I just have to figure out why the IDs of the files are out of bounds - the loading of the files seems to work perfectly. :) I'm going to keep messing around a bit with the code to see if reading different offsets from different places in file could have anything to say for the file IDs.

Also, in regards to saving memory when loading files - This is very useful, thanks again. Once I have time, I might try to redesign the loading, but right now, actually making the loading work is my highest priority. Thing is though, I would like to have some way of viewing the extracted files directly in the program using my dll when the files have been unpacked. Maybe I should just store the complete paths for all the files in the archive once they have been extracted and load them manually 1 by 1 when the user wants to see them. I don't really know yet. :)

_______________________Afr0Games

[.net] Reading binary files problem (DBPF format)

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

[.net] Reading binary files problem (DBPF format)

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines