5 replies to this topic

[radioteeth]

Hi all,

I'm working on a project at the moment that calls for a windows-style registry system, which ultimately I would like to implement in a way that stores everything as one file.

The goal is to allow for directory-style addressing of various items stored in the registry, much akin to the windows registry system. Windows itself seems to store the registry as a 'hive'. The goal here is fast read/write operations so that if anything were to happen, eg: program crashes or the machine/device unexpectedly dies or shuts down there won't be an issue of data that was in memory but didn't get written to the registry file on exit or at whatever autosave intervals, etc..

I'm not sure how to store this file, or how to interact with its contents in memory, as they will be mirrored for fast operations, and once operations are complete on the data in memory they will be written to the registry file.

Any ideas on file format? How to work with it? I thought perhaps emulating a FAT system would be interesting, or maybe just creating a blank file to act as an allocated chunk of space, and if it ever fills up just create another one, etc... I don't mind having multiple files, I was considering perhaps just using physical files, but some data are going to be very small, and cluster allocation size on different disk devices will cause these small files to be represented by 100x their size as cluster granularity isn't accustomed to such small data.


EDIT: Another requisite functionality is the ability to timestamp the database each time something is added/removed for the sake of being able to generate delta-compressed updates where if the database needs to be mirrored across two or more machines they can easily compare their most recent timestamps and the machine with the most recent up-to-date database can use the last timestamps of remote databases to generate compact updates for the remote databases to be made current.

Edited by radioteeth, 29 May 2012 - 08:20 PM.

[ApochPiQ]

Is there any particular reason you need to implement this yourself versus using an existing tool that does the same thing?

[thok]

Have a look at the HDF5 format: http://en.wikipedia....cal_Data_Format

You can store arbitrarily-structured data in a single binary file. Libs are available to for most major languages, including Java, Python, and Matlab. I'm sure there are C/C++ libs available as well (I haven't used them).

You can also use compression on the data (gzip, etc.).

h5py is a nice hdf5 library for python. I know this thread is tagged with "C", but this tutorial gives you a good sense of what you can do with HDF: http://code.google.c...h5py/wiki/HowTo . Using h5py/python might also be a nice way to prototype things (if you're already familiar with python).

EDIT: Your last requirement (in your 'EDIT') might be difficult to accomplish. You'll probably have to cook your own solution. But, still the HDF5 format should suffice for the data storage.

Edited by thok, 01 June 2012 - 08:48 AM.

[jwezorek]

How large is your fake registry going to need to be?

[DoctorGlow]

SQLite may fit the bill.

http://www.sqlite.org

[mhagain]

The registry is, at heart, just a hierarchical database, so look for some hierarchical database libraries for starters. A directory service would also do the trick here - on Windows you could probably use Active Directory Application Mode, or if you want to be portable you could grab some other lightweight LDAP directory.