Jump to content
  • Advertisement
Sign in to follow this  
flukus

How to work with a file system in memory?

This topic is 4869 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Does anyone know how to work with filesystems in memory? Say, if I open a zipped folder into memory and i wanted to access file "abc.txt" how would I access this file? I tried google a bit but it's kind of pointless when you don't know what to look for!

Share this post


Link to post
Share on other sites
Advertisement
The issues of compression and file structure are seperate. Heres something I have based my file system off of: http://www.gamers.org/dEngine/quake/spec/quake-spec34/qkspec_3.htm

Do note that the author has several discrepencies and at times cannot put together a sentence correctly. But its a step in the right direction for putting together a pack file that will let you have directories and sub-directories and files all together just like windows. For compression, I'm using ZLib (http://www.gzip.org/zlib/). The DLL interface isn't too bad, and if you google search you can find some examples on compression/decompression with it (http://www.w3.org/config/deflate.c)

Share this post


Link to post
Share on other sites
Well since this is pretty much the same topic, I have a question as well. I have my file system all spec'd out, but am stuck on one thing. When I open an existing pack file to add a new file, I'd like to be able to re-use the same file rather than create a new one and rename it to the original file name. WinRAR, WinZIP, and 7Zip all do this. 7Zip is open source but since the code isn't commented or documented very well its hard to follow. I just can't figure out how to do it other than to parse the file up to where I want to insert my new file, copy the rest of the file out to a temporary file, insert the new file, then copy all the rest of the data back over into the original file. This could be very disk intensive and slow. Ideas?

Share this post


Link to post
Share on other sites
So you have to define your own file system? Do you know of any open source implementation? I suppos it wouldn't be too bad to make my own anyway :)

Share this post


Link to post
Share on other sites
Ah, misunderstood your question. To open existing ZIP files you could use http://www.info-zip.org/

Share this post


Link to post
Share on other sites
nah, I think you got it right the first time. I want to be able to unzip a folder programmaticly (I'm using zlib) and I want to navigate the folder/read certain files while the unzipped folder is in memory.

Thinking about filesystem design is strange, thinking about file and folder structure as bytes and files is an exercise in recrsive thining ;)

Share this post


Link to post
Share on other sites
Ok, what do you guys think of the design I have so far? I decided to only allow one level of folders to save on complexity and random reading and writing to disk isn't important:


struct File
{
char[32] fileName;
long fileSize; //size of the file in bytes
long fileLocation; //the number of bytes into the data buffer the file is stored at
}

struct Folder
{
char[32] folderName;
int numFiles;
File* files[numFiles];
}

struct MountPoint
{
int numFolders;
int numFiles;
int totalFiles;
int headerSize;
Folder* folders[numFolders]
File* files[numFiles];
char* dataBuffer[];
}



Then I can calculate the size of the size of the header section of the file (where the file and folder objects are stored with:

long headerSize = (numFolders * sizeof(Folder)) + (totalFiles * sizeof(File)) + (4 * sizeof(int);

The File.FileLocation is relative to the start of the data buffer so finding the real location is done with:

fileLoc = File.fileLocation + headerSize;

I think I've thought that through pretty well, what do you guys think?

Quote:
Original post by Eps
Well since this is pretty much the same topic, I have a question as well. I have my file system all spec'd out, but am stuck on one thing. When I open an existing pack file to add a new file, I'd like to be able to re-use the same file rather than create a new one and rename it to the original file name. WinRAR, WinZIP, and 7Zip all do this. 7Zip is open source but since the code isn't commented or documented very well its hard to follow. I just can't figure out how to do it other than to parse the file up to where I want to insert my new file, copy the rest of the file out to a temporary file, insert the new file, then copy all the rest of the data back over into the original file. This could be very disk intensive and slow. Ideas?


Wouldn't that require an insert operation for whatever filesystem the OS uses? I vaguely remember hearing about something like that being available in reiser4!

Share this post


Link to post
Share on other sites
Well, it should certainly work, but you may have over complicated things a bit. I'd have to mull over the details of you design to discover any pros/cons, so instead I'll show you what I came up with and you can compare it for yourself.


// Pack Entry
struct PackEntry_T
{
PackEntry_T() { memset(this, 0, sizeof(PackEntry_T)); }

short type; // Type of entry: Folder(0) or File(1).
unsigned char string[54]; // Name of the entry, 52 chars, padded with '\0'.
long size; // Uncompressed size of the entry in PACK file if a file, or number of entries if a folder.
long csize; // Compressed size of the file.
};


Thats it. Using this structure I can have any number of levels of directories and any number of files. If I set an entry to type folder and write it out, it will tell me that the next 'x' number of entries read belong to that folder. That could include another folder inside of which could be more folders. If I want to insert a file into a folder, I read through the file until the correct folder is found, increment its file count, then immediately write in the new file.

The first thing in my file is always a root folder, under which i can keep track of everything that is going on. This is a permenant folder that cannot be removed.

You may notice I'm not keeping track of file offsets within the pack. This is because I'm writing things out in the "header-buffer-header-buffer" fashion instead of "header-header-buffer-buffer". This lowers the maintanence I need to do when writing/removing.

Your structs are 4 byte aligned which is good, if they weren't they would get padded by the default compiler options (in VS anyhow) and when trying to parse the file by the size of structs you'd run into problems.



If anyone read all that, let me know if you see any issues with my design, as I think I have thought it out fairly well too :)

Share this post


Link to post
Share on other sites
Quote:
Wouldn't that require an insert operation for whatever filesystem the OS uses? I vaguely remember hearing about something like that being available in reiser4!


Couldn't find this feature for reiser in just a couple min of searching, but I'm on a windows box anyhow. Any idea if there is a similar feature available (whatever it does?)

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!