Does anyone know how to work with filesystems in memory? Say, if I open a zipped folder into memory and i wanted to access file "abc.txt" how would I access this file? I tried google a bit but it's kind of pointless when you don't know what to look for!

The issues of compression and file structure are seperate. Heres something I have based my file system off of: http://www.gamers.org/dEngine/quake/spec/quake-spec34/qkspec_3.htm

Do note that the author has several discrepencies and at times cannot put together a sentence correctly. But its a step in the right direction for putting together a pack file that will let you have directories and sub-directories and files all together just like windows. For compression, I'm using ZLib (http://www.gzip.org/zlib/). The DLL interface isn't too bad, and if you google search you can find some examples on compression/decompression with it (http://www.w3.org/config/deflate.c)

Well since this is pretty much the same topic, I have a question as well. I have my file system all spec'd out, but am stuck on one thing. When I open an existing pack file to add a new file, I'd like to be able to re-use the same file rather than create a new one and rename it to the original file name. WinRAR, WinZIP, and 7Zip all do this. 7Zip is open source but since the code isn't commented or documented very well its hard to follow. I just can't figure out how to do it other than to parse the file up to where I want to insert my new file, copy the rest of the file out to a temporary file, insert the new file, then copy all the rest of the data back over into the original file. This could be very disk intensive and slow. Ideas?

So you have to define your own file system? Do you know of any open source implementation? I suppos it wouldn't be too bad to make my own anyway :)

Ah, misunderstood your question. To open existing ZIP files you could use http://www.info-zip.org/

Or take a look at ZLib as well.

- Drew

nah, I think you got it right the first time. I want to be able to unzip a folder programmaticly (I'm using zlib) and I want to navigate the folder/read certain files while the unzipped folder is in memory.

Thinking about filesystem design is strange, thinking about file and folder structure as bytes and files is an exercise in recrsive thining ;)

Ok, what do you guys think of the design I have so far? I decided to only allow one level of folders to save on complexity and random reading and writing to disk isn't important:



Then I can calculate the size of the size of the header section of the file (where the file and folder objects are stored with:

long headerSize = (numFolders * sizeof(Folder)) + (totalFiles * sizeof(File)) + (4 * sizeof(int);

The File.FileLocation is relative to the start of the data buffer so finding the real location is done with:

fileLoc = File.fileLocation + headerSize;

I think I've thought that through pretty well, what do you guys think?

Quote:Original post by Eps
Well since this is pretty much the same topic, I have a question as well. I have my file system all spec'd out, but am stuck on one thing. When I open an existing pack file to add a new file, I'd like to be able to re-use the same file rather than create a new one and rename it to the original file name. WinRAR, WinZIP, and 7Zip all do this. 7Zip is open source but since the code isn't commented or documented very well its hard to follow. I just can't figure out how to do it other than to parse the file up to where I want to insert my new file, copy the rest of the file out to a temporary file, insert the new file, then copy all the rest of the data back over into the original file. This could be very disk intensive and slow. Ideas?

Wouldn't that require an insert operation for whatever filesystem the OS uses? I vaguely remember hearing about something like that being available in reiser4!

Well, it should certainly work, but you may have over complicated things a bit. I'd have to mull over the details of you design to discover any pros/cons, so instead I'll show you what I came up with and you can compare it for yourself.

// Pack Entrystruct PackEntry_T{ 	PackEntry_T() { memset(this, 0, sizeof(PackEntry_T)); }	short type;                  // Type of entry: Folder(0) or File(1).	unsigned char string[54];    // Name of the entry, 52 chars, padded with '\0'.	long size;                   // Uncompressed size of the entry in PACK file if a file, or number of entries if a folder.	long csize;                  // Compressed size of the file.};

Thats it. Using this structure I can have any number of levels of directories and any number of files. If I set an entry to type folder and write it out, it will tell me that the next 'x' number of entries read belong to that folder. That could include another folder inside of which could be more folders. If I want to insert a file into a folder, I read through the file until the correct folder is found, increment its file count, then immediately write in the new file.

The first thing in my file is always a root folder, under which i can keep track of everything that is going on. This is a permenant folder that cannot be removed.

You may notice I'm not keeping track of file offsets within the pack. This is because I'm writing things out in the "header-buffer-header-buffer" fashion instead of "header-header-buffer-buffer". This lowers the maintanence I need to do when writing/removing.

Your structs are 4 byte aligned which is good, if they weren't they would get padded by the default compiler options (in VS anyhow) and when trying to parse the file by the size of structs you'd run into problems.



If anyone read all that, let me know if you see any issues with my design, as I think I have thought it out fairly well too :)

Quote:Wouldn't that require an insert operation for whatever filesystem the OS uses? I vaguely remember hearing about something like that being available in reiser4!

Couldn't find this feature for reiser in just a couple min of searching, but I'm on a windows box anyhow. Any idea if there is a similar feature available (whatever it does?)