Jump to content
  • Advertisement

Archived

This topic is now archived and is closed to further replies.

FxMazter

designing a FileSystem ! :)

This topic is 5270 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello everyone! This is the first time I''m trying to make a FileSystem. And I''ll spitt it out... I got no clue about where to start :/ I''ve been thinking about some stuff: 1. It has to replaceable, so I can change a filesystem... 2. I want to have a structure that let''s adding new filetypes along with the engine, such as bmp files or pak files. 3. It should be easy to use. So with those things in mind I started up thinking about the design: 1. Working with interfaces, will make it easy to change the FileSystem implementation. class IFile; //IFileOperations is an interface for IO write/read functions //for easily changing Platform class IFileOperations; //IFileCreator will create the correct IFile for a specific //filetype, like .bmp or .pak etc... class IFileCreator; class IFileSystem { public: virtual IFile* OpenFile(const char* sPathName) = 0; virtual void SetIODrivers(IFileOperations* pIODriver) = 0; virtual void RegisterFileType(const char* pFileEnding, IFileCreator* pCreator) = 0; //... What more would be useful ... // ? }; That will enable changing IO drivers for Win or Unix etc... And also adding new FileTypes. Now before looking on the IFile, we need some sort of pathfinding system that will check directorys, look for files etc...: class IPath { public: IPath(const char* sPathName) { //... } //Is needed for Finding Directorys in differend Platforms static void setFileOperations(IFileOperations* pFileOperations) { m_pFileOperations = pFileOperations; } virtual bool addPath(IPath newPath) = 0; virtual bool addFile(IFile* pFile) = 0; virtual bool addFile(IPath path, IFile* pFile) = 0; virtual bool contains(IPath findPath) = 0; virtual bool contains(IFile* pFile) = 0; virtual IPath* getPath(const char* sPathName) = 0; virtual IFile* getFile(const char* sFileName) = 0; virtual const char* getPathName() = 0; private: private static IFileOperations* m_pFileOperations; }; Now IPath implementations should give enougth operations to easily iterate through the filesystem looking for files or directorys. For eg: class CDirecotry : public IPath { //... }; Ok, lets look at the IFile: //BaseClass for headers of different filetypes... //All IFileHeader''s will have default data as: //1. FileTypeId, eg. BMP_HEADER, JPG_HEADER, PAK_HEADER class IFileHeader; class IFile { public: virtual void Save() = 0; virtual void ReadFileHeader(IFileHeader* pHeader) = 0; virtual IFileHeader* getFileHeader() = 0; virtual IPath* getPath() = 0; virtual void* getFileData() = 0; //... what more could be useful ?? }; OK, I think this should work just fine for a quite ok filesystem. But I''m having trouble with the Save() / ReadFileHeader(IFileHeader* pHeader) stuff. It works just fine if the given IFile is "real" file in a Directory. Then I can just write the file to disk, overwriting the old one. But what if the given IFile is not "real", such as if it''s in a .PAK file? Then I can''t just write it to disk, since it''s actually the .PAK file that should write the data. So then it should redirect the Save call to the .PAK file! But, the thing is that .PAK files are just an IPath like this: class IPakFile : public IFile, public IPath { //... } That way at startup I can just open all PAK files and create a TREE system with IPath''s. And later when searching for a file the PAK files will work just as an IPath, that is just as a Directory... But How is a single IFile supposed to know if it should Save(), that is write the data directly to disk or REDIRECT the call to a specific PAK file? I mean, the IFile does not know that it comes from a IPakFile. All it knows is that it comes from a specific IPath, which could be a CDirecotry or a IPakFile. I have been thinking of redirecting ALL calls from the IFile to the parent IPath, where the IFile is contained. And then the IPath could redirect the call to it''s parent IPath etc... Until the call arrives at a IPath that is actually an CDirectory, which will write the file to disk, or the call could arrive at an IPakFile, which will write the file to ITSELF! First off, Big thanks for just reading this long post I would really appreciate some ideas on how to do this in some other way, and I would especially appreciate some info on how the IFileOperation''s should look like. - I was thinking about something like WriteFile(...), FindFile(...), CreateDir(...), CreateFile(...). But I''m not sure how those functions should look to make it possible to implement in different Platforms. Thank you!

Share this post


Link to post
Share on other sites
Advertisement
Yes, now I have

But I would still rather make my own. Since this engine is not
all about finishing it fast, but rather learning along the way
and maybe use it as a demo in future job applications

But thx anyway

Well, I downloaded the boost filesystem. And according to the
docs they also have some Path system. But I couldn''t get out much
more of it than that. :/

Share this post


Link to post
Share on other sites
The way I designed mine was to have a class which represented a "handle" to a file loaded into the system, inside this class is a variable which stores where the file originated from - disk/zip/pak etc so when I add a file into the system I can then call ReadFile() (inside the file system class) passing the file handle returned from the Open() method as the paramter.

There''s probably better ways to do it but that way works for me and it saves you having to worry about where the file originated from - you call the same method to read the file from an archive or from disk.

if you like I can email you the source to my VFS if you wanna check it out as it''s quite simple as should be easy to go through to give you some ideas on how to get started.

Share this post


Link to post
Share on other sites
quote:
Original post by FxMazter
Hello everyone!


Hello!
quote:

1. It has to replaceable, so I can change a filesystem...


So, I have no idea what you mean here.
quote:

2. I want to have a structure that let''s adding new filetypes
along with the engine, such as bmp files or pak files.


I don''t think the filesystem needs to care about filetypes. You could just the filename extension, or a mimetype attribute, to determine the type of a file.
quote:

3. It should be easy to use.


Easy to use is easy. Actually implementing it will be the tricky bit.
quote:

That way at startup I can just open all PAK files and create
a TREE system with IPath''s. And later when searching for a file
the PAK files will work just as an IPath, that is just as a
Directory...

But How is a single IFile supposed to know if it should Save(), that is write the data directly to disk or REDIRECT the call to
a specific PAK file?

I mean, the IFile does not know that it comes from a IPakFile.
All it knows is that it comes from a specific IPath, which could
be a CDirecotry or a IPakFile.

I have been thinking of redirecting ALL calls from the IFile
to the parent IPath, where the IFile is contained. And then
the IPath could redirect the call to it''s parent IPath etc...
Until the call arrives at a IPath that is actually an CDirectory, which will write the file to disk, or the call could
arrive at an IPakFile, which will write the file to ITSELF!

First off,

Big thanks for just reading this long post

I would really appreciate some ideas on how to do this in some
other way, and I would especially appreciate some info on how
the IFileOperation''s should look like. - I was thinking about
something like WriteFile(...), FindFile(...), CreateDir(...),
CreateFile(...).

But I''m not sure how those functions should look to make it
possible to implement in different Platforms.

Thank you!



I think the first thing to realise is that you''re trying to code two different systems here: a portable interface to the native filesystem, and a virtual filesystem that uses the portable native interface and archive files, such as PAK (or ZIP or whatever).

You should realise that writing to a PAK file is not going to be a simple operation. PAK files store their content contiguously, so if you lengthen a member of the archive, it will need to relocated to the end of the archive. Eventually, you''ll need to compact it or it''ll eat all the space on your disk.

Most games that use a VFS have multiple input filesystems, but only one output, usually a user-specific directory in the native filesystem.

There are a couple of important advantages to this. Firstly, if the archives you distribute aren''t modified, it''s safe for you to distribute updated versions that don''t mess up the user''s save data. Secondly, if the game is being used by multiple users, then each user can have their own copy of the save data without interfering with other users and needing privileges to play the game.

Obviously, when you''re actually making the game, you''ll need to be able to write to the ''main'' filesystem. For efficiency''s sake, you''ll almost certainly want to use the native filesystem when developing and put it in a PAK for distribution.

I don''t think that''s a good reason to leave out PAK-writing support, though.

The first thing you need to deal with is the actual filesystems. For these, you implement Filesystem, Directory, DirectoryEntry, File, and what ever other classes you need to describe the filesystems.

You need to define these for each actual filesystem.

The next thing is the virtual filesystem. The virtual filesystem looks, from the outside, just like another implementation of the virtual classes you defined above. On the inside, of course, it behaves entirely differently.

Each virtual directory contains a list of the actual files from each filesystem, a list of virtual subdirectories.

Because the files stored in the VFS are the actual files, there''s no need to do anything special with an IFile you get from the VFS.

Extra magic

Many OSes support a filesystem notification mechanism that allows your program to be notified when a file or directory is modified. The native filesystem could respond to such notifications and update the VFS.

Real filesystems support fragmented files, thus allowing efficient modification of existing files. You could create an archive format that also supports fragmented files.

You could implement a filesystem that grabs files remotely via HTTP/FTP.

I said that the filesystem doesn''t need to know about filetypes, but you could make an exception. Allow a file to be marked as a patch of the existing file, and make the VFS transparently apply the patch, thus allowing small updates to large files to be small downloads.

Create a filesystem that makes information about the state of the game available as files.

Share this post


Link to post
Share on other sites
Take a look at this tutorial about 'Virtual file systems' . It may help..

I used this tutorial to implement a "pak" file system using zlib for compression/decompression and a templated resource manager that read/wrote to the archive.. A hell'uva lot coding behind, but it made it very simple to deal with resources, when it was finished..well.. it actually never got to that point

[edit] Posted it by accident...

[edited by - Rickmeister on June 8, 2004 3:59:25 AM]

Share this post


Link to post
Share on other sites
Whats with the I before your class names. I''m sorry and I''m a newbie so I don''t understand it. Though I see it in some other threads. Does this designate your class hiarchy as special in some way? Will it behave differently when implemented? I dont really get it. Thanx for the explanation.

Share this post


Link to post
Share on other sites
The I usually stands for interface.

It indicates that the class is an abstract baseclass whose only use is to define an interface to different kinds of the same thing

in this case, you have an IFile .. and interface to a file

because PAKFile and NativeFile both derive from IFile, you can treat them exactly the same, even through they are stored and maintained differently

The place where i''ve seen it used most is in the COM system, where everyting is derived from IUnknown

If you ever look at directX you can''t avoid it ;-)

HTH,
Willem

Share this post


Link to post
Share on other sites
Wow, fantastic explanation Willem, thank you :] The original author states that he may someday use this code in a resume. I program in the same way all the time using abstract classes, and oop hierarchy. Will I have a better looking resume if I start turning all abstract systems into interfaces? Such as -> class Entity {method()=0;} to class IEntity {method()=0;}?

Share this post


Link to post
Share on other sites
I only bother with ''I'' if i''m using COM. I''d concentrate on making your code read-able and bug-free, thats what most people are looking for. Your naming convention isn''t really going to help too much with this.

Share this post


Link to post
Share on other sites

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!