Jump to content
  • Advertisement
Sign in to follow this  
Basiror

pakfile implementation *suggestions wanted*

This topic is 4675 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi I am about to implement a pakfile system currently my design is as follows //file communicates with a file_system singleton //whenever I open a file I search for the file in the pakfile //if successful I open the pakfile and read in the entire file into a buffer //otherwise I search for the file on the disc if I find it I buffer it as well //otherwise I return FALSE; //the read write operations only access the buffered data //maybe I implement a garbage collector for allocated memory blocks for reusing //them later class file; //file_system holds a list of all pak files and handles file<-->pakfile //communication class file_system; //pakfile holds a file table with the filename seperated from the path //each file table entry stores filename and relative path //e.g.: usr/game1/tex1/texture.tga //filename: texture.tga //path: usr/game1/tex1 //when i load class pakfile; I wonder about whether a pakfile will ever contain more then 65536 files at once I haven t seen this in any game yet although I am currious about this because i want to pack the file positions into a 32bit integer 16bit for the pakfile index 16bit for the file index inside the pakfile

Share this post


Link to post
Share on other sites
Advertisement
Quote:
Original post by Basiror
Hi I am about to implement a pakfile system currently my design is as follows

//file communicates with a file_system singleton
//whenever I open a file I search for the file in the pakfile
//if successful I open the pakfile and read in the entire file into a buffer
//otherwise I search for the file on the disc if I find it I buffer it as well //otherwise I return FALSE;
//the read write operations only access the buffered data
//maybe I implement a garbage collector for allocated memory blocks for reusing //them later
class file;

//file_system holds a list of all pak files and handles file<-->pakfile //communication
class file_system;

//pakfile holds a file table with the filename seperated from the path
//each file table entry stores filename and relative path
//e.g.: usr/game1/tex1/texture.tga
//filename: texture.tga
//path: usr/game1/tex1
//when i load
class pakfile;

I wonder about whether a pakfile will ever contain more then 65536 files at once
I haven t seen this in any game yet although I am currious about this because i want to pack the file positions into a 32bit integer
16bit for the pakfile index
16bit for the file index inside the pakfile


If you're only using 16 bits to store file positions (I assuming this what mean by file index) your limiting yourself. You won't beable to store file locations beyond 65636 bytes. I would recommend using a 32 bit integer for file locations.

Share this post


Link to post
Share on other sites
Still I suggest to not limit yourself to 65k files, doesn't
cost much to take 32bit instead of 16 bit.

And I would give external files priority over files inside the
archive - nice for patching (instead of users having to download
your 500mb file, they can overide the 1 new file they need -
or add custom graphics without changing the archive).

And instead of making a new format, consider taking .zip (google
for minizip, zzip) - easy, comfortable to create & open fileformat.

/R

Share this post


Link to post
Share on other sites
erm I think i explain it again

int32 file_system::find_file(const string& filename)
{
return (pakfileindex<<16 | fileinpakfile_index)
}

this way i want to return the index of the pakfile and the index of the desired file inside the pak file


and i don t plan to use 500mb sized pakfiles either

i have a main and a mod folder


when loading the pakfiles
I search the mod folder at first to load the latest files at first
sort this by last modification date of the pak file
then I search the main game path for the original pak files


the map editor paks all external files used in the map into the pak file
the engine itself will use external files if it couldn t find a certain file in on of the pakfiles

this is how i plan to implement it

this way i want to keep map downloads and mods simple all you have to do is load the pak file and everything works fine without missing files and only those files used will be submitted

one problem that still exists is overwriting of textures for example
which might influence the look of a map so i could either to a CRC check if the client uses different textures or I need to introduce a strick naming convention
e.g.: that custom textures used in certain maps have to look like this
<mapname><texturename><extension>

same for all other custom resources
this might solve some of the problems the q3 pak file system introduces
in call of duty for example you find a lot of servers with custommaps and nobody has a clue which maps/paks to download to get everything running

i also won t implement this unmodified client feature it introduces more bugs than benefits
a strickt file checking routine which checks all files used in a certain map should guarentee the correctness on the client
the resources loaded from inside the engine will also be checked at connection time

Share this post


Link to post
Share on other sites
You could add an header to your archive.

The header contains a serialized version of the file map (name, offset, size)

So when you load the pak file simply rebuild the map (the key could be the filename)

When you need a file search for the name in the map and seek the archive file using the associated offset.

This is how I implemented my archive; I also compress each file with zlib.

Share this post


Link to post
Share on other sites
One thing to consider would be to allow reading a file from the archive without needing to decompress/read it entirely in memory. If you have a big music file for example, it would make more sense to stream it directly from the archive than from memory. Just a suggestion. I do that with my format, the user is free to either load the file in memory or use it directly, or even use a buffer layer.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
A std::vector is going to have to hold 65536 entries for EVERY packfile just so packfile #2 can start at index 65536 even if packfile #1 has only 1 file in it!
I think you should rethink your design a little to save some space; why not just have a separate vector for each packfile you open, or a vector of vectors?

Share this post


Link to post
Share on other sites
erm no?

you can get the packed indices again when you need em

fileindex = 0x0000FFFF & index;
pakindex = index >> 16;

no reason to store a 2^16 sized vector



persil do you use fmod for audio support ?
I plan to use fmod but I don t know if it supports streaming

Share this post


Link to post
Share on other sites
Quote:
int32 file_system::find_file(const string& filename)
{
return (pakfileindex<<16 | fileinpakfile_index)
}

Is the only reason you want to return a packed 32-bit value so that it fits in a single integer? In this case you could just use a std::pair<unsigned, unsigned> and not limit yourself to 65k resources.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!