Home » Community » Forums » » Creating a PAK File Format
  Intel sponsors gamedev.net search:   
[Control Panel] [Register] [Bookmarks] [Who's Online] [Active Topics] [Stats] [FAQ] [Search]

Add Forum to Favorites |  Send Topic To a Friend | View Forum FAQ | Track this topic

Page:   1 2 »»

 Last Thread Next Thread 
 Creating a PAK File Format
Post Reply 
Never save the cipher value inside the file!!
if you want a safe PAK of course



 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Nice article...

Tough I would have put the file table at the end of the file. That way wou dont have to rewrite the whole PAK-file everytime you add a new file to it (no need to move the data to make room for the new entry in the file table), just append the file table after the data part. Makes sence when the table part i 16Kb and the data part is 300Mb, you chose which one to rewrite. The header should hold the address of where the file table and the data start, and how long each is. The file table is just as it is, with offset and length of data. Blabla etcetc.

Of course, all this isn't needed if all you ever do is read from your pak-file.

 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

On a side note, if you make the decision to memory-map your PAK file, you may want to ensure that each file begins on a 4k boundary so that the Windows paging system can more efficiently bring in your files.

 User Rating: 1099   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

I like the article, though the comments above are valid. The next stage would be to create a virtual File class so you can abstract the difference between a file in the Pak file and a normal file - to allow for files to be overidden ala Half-Life(and other such engines) style.

Perhaps we could expand the system to have a folder system within the directory system to allow for better management of the files(aviod name clashes and better organization before compilation)

 User Rating: 1263   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

This (responding to last comment) was covered in Chris Hargrove's articles in Code on the Cob Part 4 and part 6.

Coded in C, but I'm sure people who are interested could take the code in this article and combine with the principles of these two older articles.

Oli


All the best problems start with C

Manta-X - A 3D arcade shoot-'em-up [In development]

 User Rating: 1902   |  Rate This User  Send Private MessageView ProfileView Journal Report this Post to a Moderator | Link

This is my packformat, which i use in my engine.

[pack header]
[filter header 1]
.
.
.
[filter header n]
[dir header 1]
.
.
.
[dir header n]
[file header 1]
[file data 1]
[data descriptor 1]
.
.
.
[file header n]
[file data n]
[data descriptor n]


A. Pack Header

id 16 bytes
filename len 4 bytes
filename (variable size)
version 4 bytes
number of filters 4 bytes
number of dirs 4 bytes
number of files 4 bytes
blocksize 4 bytes

ToDo:
last mod time
last mod date

B. Filter Header

id 4 bytes
name len 4 bytes
name (variable size)

ToDo:
needed version 4 bytes

C. Dir Header

name len 4 bytes
name (variable size)
id 4 bytes
parent dir id 4 bytes

D. File Header

name len 4 bytes
name (variable size)
dataoffset 4 bytes
dir index 4 bytes
hash 32 bytes

E. Data Descriptor

uncompressed size 4 bytes
compressed size 4 bytes
used filters ? bytes


So this woule represent one file:
File Header
File Data
Data Descriptor

What do think about it?

 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Just out of curiosity...

I noticed that ther was a szUniqueID in the pak.h which was later assigned a random number in pak.cpp. I was wondering what this information was used for or why it was put there. I tried looking at where it would be used throughout the program and couldn't find anything. Is this kind of like a distraction thing?

And my other question:
Is it just me or does the dummy node serve no purpose? I tried tracking it after it was created; the m_FileTable was set to dummy, but later in the GenerateHFT() I noticed that m_FileTable was set to the tempEntry node. And in workoutOffsets() the current pointer was set to m_FileTable. Does anyone know what the dummy node is for? & did I overlook something?


Thanks,
Hovig

[edited by - Toonkides on September 7, 2003 9:37:42 AM]

 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Hello, I'm the author of this article (using a newish account) and I would just like to say, firstly, thank you for your polite, informative and inquisitive responses to this article.

In response to Toonkides - The dummy node is inserted as the head of the linked list to avoid problems that can occur when inserting or deleting nodes with an empty list. This way the head of the list never needs to be changed. I believe it is good practice to do so. It is not actually written in to the PAK file in this application.

Also, the unique ID is not actually used by the PAK program itself but I was thinking it could be something the application that utilises the PAK file checks.

Ray

 User Rating: 1000   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

I noticed that you were using the "BOOL" and "DWORD" typedefs.
Since you are programming this in C++, you must know that there is already a bool type in the language. As for dword, you could simply use an unsigned int. Making your code independent from the windows headers will only make it better.

I also saw this function prototype:
BOOL CreatePak( char* Path, char* Output);

And I believe you should be using a const char* for the path, since there is no reason to modify the input string.

[edited by - Max_Payne on September 7, 2003 1:13:53 PM]

 User Rating: 1438   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

bool is evil. it used to be one-bit wide, then one-byte.

i use an unsigned char instead.

ints are evil too - they're 16-bit on 16bit platforms, 32 on 32, and 64 on 64. use short and longs - garenteed to be 16 and 32-bits respectivly.

typedef signed char int8;
typedef unsigned char uint8;

typedef signed short int16;
typedef unsigned short uint16;

typedef signed long int32;
typedef unsigned long uint32;

typedef uint8 boolean;

that's what i use anyway

 User Rating: 1168   |  Rate This User  Send Private MessageView ProfileView Journal Report this Post to a Moderator | Link

how can i convert the data which is in the pak to a directdrawsurface.can anyone show a load function example.

 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

sTratejist: I assume you want to read a bitmap from the PAK to a surface?? Just create a surface thats the size of your bitmap in the PAK file and lock the surface and then fread the bitmap into a buffer, then memcpy the buffer to your surface, then Unlock the surface. a little complicated.

Denny.

 User Rating: 1019   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Just some question about your code. I'm a nubie.

How come, when you Encrypt and write each file table entry to the PAK file, you copy the whole file table to another buffer??
this is the code:
// Create a BYTE pointer for byte access to a file table entry

BYTE* Ptr = NULL;

// Create a BYTE array the same size as a file table entry

Ptr = new BYTE [sizeof(sFileTableEntry)];

// Copy the current file table entry in to the byte array

memcpy( Ptr, Current, sizeof(sFileTableEntry) );


when you could just do this (untested code):
   
BYTE* Ptr = (BYTE*)Current;

BYTE b;
for (.....)
{
    b = Ptr[ i ];
    //Do encryption stuff and write to file.

}


Also, when you are writing the files to disk (PAK), why are you opening the PAK file for every file written to the PAK file?? Why can't you keep it open until all the files are written? they should be sequentially written right?

I'm always asking questions....nosy me.

Denny.

[Edit]: more questions

[edited by - brain21 on September 7, 2003 9:32:24 PM]

 User Rating: 1019   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

brain21: thanks for your reply but a have one more question.i didn't work on files before and i don't know how will i fread the bitmap to a buffer from pak file.for example i create a pak file from 2 bitmaps and i want to read the second one.how can i get the file information from header.if you can help with code i will be happy :D.

 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

sTratejist - You could add a public function to the class to load the header and file table in to the class structure. Say you used this load function to read in the header and file table from the specified PAK. Please not it is 00:40 in the morning so there may be some errors in this code ... Something like this:

BOOL CPakFile::LoadPakHFT( char* FileName )
{
	// Declare local variables

	FILE*			InputStream;

	// Error check

	if( strlen(FileName) == 0 ){ return FALSE; }
	
	// Open the specified file

	InputStream = fopen( FileName, "rb" );
	if( !InputStream ){ return FALSE; }

	// Read in the file header

	fread( &m_Header, sizeof(sPakHeader), 1, InputStream );

	// Read in the file table

	for( DWORD i = 0; i < m_Header.dwNumFTEntries; i++ )
	{
		// Create a BYTE array the size of a file table entry

		BYTE* Ptr = NULL;
		Ptr = new BYTE [sizeof(sFileTableEntry)];

		for( DWORD z = 0; z < sizeof(sFileTableEntry); z++ )
		{
			BYTE Temp = 0;

			// Read a BYTE at a time

			fread( &Temp, sizeof(BYTE), 1, InputStream );

			// Perform decryption using the caesar cypher (reversed to decrypt)

			if( m_Header.bCypherAddition == TRUE )
				Temp -= m_Header.iCypherValue;
			else
				Temp += m_Header.iCypherValue;

			// Add to the BYTE array

			Ptr[z] = Temp;
		}

		// Create the file entry to add to the list

		sFileTableEntry* NewEntry = new sFileTableEntry();

		// Copy the contents of the BYTE array to the entry

		memcpy( &NewEntry, Ptr, sizeof(sFileTableEntry) );

		// Add this to the linked list, file table member variable

		NewEntry->Next = m_FileTable;
		m_FileTable = NewEntry;

		// Delete the temporary BYTE array

		delete [] Ptr;
	}

	// Close the file stream

	fclose( InputStream );

	return TRUE;
}


You could generate a "Find" function that takes the file name in the PAK you want to use as a parameter. This function could return type sFileTableEntry if it is found. This way you would be able to have the files offset in the PAK as well as its size ... You set the position in the PAK file in the same way that was demonstrated in the Create() function and read the data a BYTE at at time in to a buffer.

[edited by - __Daedalus__ on September 8, 2003 7:55:44 PM]

 User Rating: 1000   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

at last i did what i want.thanks for your help.

 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

I have updated the PAK source code. The PAK class that is used in the code
now includes functions to load a header and file table from an existing PAK
file, extract all contained files within a specified PAK file, extract a
specific file from the specified PAK file by name and also to clear out the
contents of the PAK class (for use in programming with it). You can download
the new source code from my website. The URL is:

http://www.bytefoundry.co.uk/downloads/pak%20article%20source.zip

The included compile tool now has the facility to extract an entire PAK as
well as the compilation features it had before. If you have any trouble
extracting an old PAK file then you should recompile it using the new tool
(I don't think anything significant has changed in the compilation process
though so everything should be readable).


 User Rating: 1254   |  Rate This User  Send Private MessageView ProfileView Journal Report this Post to a Moderator | Link

I was working on a game and I have two graphic files and I will add sound later. My question is this: The pak file is good for storing all media into one file, but how do I make the functions which draw the bmps from a certain position in the pak? The parameter seems to only be a filename. Thanks

 User Rating: 1015   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Is extracting the only way to use things like lua scripts? You have to pass a filename. If this is right then hackers can still get easy access to the files when you extract them.

 User Rating: 1341   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Hey... Im trying to extract some files froma psuedo PAK file...
Its from an old game made by FunLabs. The file extension is FUN.
But upon looking intor the hex of the game EXE I notted that it runs under the PAK format. pak.cpp was clearly noted. I'm trying to use the pak compiler to extract the FUN files. It's not seeming to work. But i don't get any error messages. Where does the Pak compiler extract too? anyone help? or care to look at a samle of this file format to see how closely it relates to the PAK since it obviouslty uses a pak extraction method.

if i cannot get PAK to extract.. I also noted that the game uses LUA scripts... I can add an additional script that (if possible) that would Copy the contained files to a set destination? can LUA files be written to copy and paste essentially in an automated manner?

ideas on either approach please help.

 User Rating: 1015    Report this Post to a Moderator | Link

It seems like you can load the data from files in a PAK, but this doesn't actually unpack the files into a specific directory.

I suppose you could simply take the data you've loaded into your program and write it once again into a directory, but without a way of automatically deleting it upon program shutdown, those files will exist, unpacked and exposed, after the program has shut down.

The reason I need to unpack files and place them in directories (not just load the information into the program) is so I can use the D3DXCreateTextureFromFileEx() function to load bitmaps as textures. The function only takes filenames as a paramater, not an array of data in my program.

Any suggestions as to how I can use PAK files to mass-store and safeguard my game media, while still being able to use these kinds of functions?

Thanks.

-Gauvir_Mucca

 User Rating: 1033   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

http://www.bytefoundry.co.uk/downloads/pak%20article%20source.zip

is no longer working it seems :(

does anyone have the file ??

a link to it or perhaps can email me it to admin(at)freplay.dk ?

reasons is i want the code with added extract method :)

thanks

 User Rating: 940   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Awesome, thank you for the quick email :)

 User Rating: 940   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Okay, very interresting, i now added this to my game system, and added a way to get data into directx ect from the file.

however, i would like to have :

add file to current archive (woudnt be that hard)
and update file in archive (wich i got no clue how to do so its fast and reliable).

Anyone got any idea's atleast about the update a file ?

Possibly without having to rewrite the intire pak structure and data ?

Thanks :)

 User Rating: 940   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link
Page:   1 2 »»
All times are ET (US)

Post Reply
 Last Thread Next Thread 
Forum Rules:
You may not post new threads
You may post replies
You may not edit your posts
You may not use HTML in your posts
Jump To:
Administrative Options: