Jump to content
  • Advertisement
Sign in to follow this  
FenixRoA

C# Database/Dataset

This topic is 3709 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello again GameDev, I want to store a bunch of pictures in a database/dataset. It's possible that this database will store all of the pictures/photos on someones computer, so the size of the database can easily balloon up. At the moment there is no need to transfer the data or any part of it over a network. From what I understand (and I am a NEWBIE as far as database/dataset programming and the difference between them) if I want to use a dataset, I would either need to save the dataset into seperate files, or load the entire dataset (which may have ballooned to the size of a few gigabytes), use it and then save the entire dataset. Is this true? Would I be forced to do that? If so, would using an access database be any better? I've looked up a bit about this, and I figured out how to save the dataset to a binary file (without it being all XML text), but I can't find any articles that explain to me exactly what I need. Maybe I just don't know the keywords I need to google. Your help is very much appreciated, ~FenixRoA

Share this post


Link to post
Share on other sites
Advertisement
Quote:
Original post by FenixRoA
From what I understand (and I am a NEWBIE as far as database/dataset programming and the difference between them) if I want to use a dataset, I would either need to save the dataset into seperate files, or load the entire dataset (which may have ballooned to the size of a few gigabytes), use it and then save the entire dataset.


A dataset is a snapshot of part or all of a database. And yes, if you have a single dataset that is gigabytes in size, then yes, when you load(deserialize) it, the entire dataset will be loaded into memory. If your dataset causes your program to exceed the 2GB user address space limit, you will get an OutOfMemoryException.

Quote:
At the moment there is no need to transfer the data or any part of it over a network.


If the dataset resides on the computer with the images, why can't you store the path to each image instead of the image itself in your dataset? This would significantly decrease the size of your dataset, allowing you to have one dataset that can be loaded into memory (assuming you aren't referencing hundreds of thousands of images).


Share this post


Link to post
Share on other sites
Thank you for the reply CyanSoft.

I had considered this, however I want to implement a limited form of privacy. Firstly I want the pictures to not be searchable from outside my application (I will do this by encrypting them and putting them inside the application). Further more I don't want my application to have a file for each picture (as this grants even the simple user the ability to figure out how many pictures are stored in the program, and can even selectively delete images).

The images will be stored in the database in one of two ways. So called 'private images' will have their data encrypted and stored in the database. Other images will just have their path stored within the database.

I suppose what I'm actually asking is if there is any way to have an entire database that just loads necessary parts to memory.





~FenixRoA

Share this post


Link to post
Share on other sites
Quote:

The images will be stored in the database in one of two ways. So called 'private images' will have their data encrypted and stored in the database. Other images will just have their path stored within the database.


You may not need to encrypt them, instead have different levels of access usernames/passwords to the database itself.

Quote:

I suppose what I'm actually asking is if there is any way to have an entire database that just loads necessary parts to memory.
~FenixRoA


There is alway a way, but I'm unclear as too why you need to do this. (can you explain a little more?)

I've implemented a system in .NET C# that stores images into a database, for much the same reasons you mentioned above, it also really helps in not having to work out a file system for storing images. My system pulls images out of the database at a good speed without any issues, so I'm a little confused!

Share this post


Link to post
Share on other sites
Well again, the database may become rather large.

If my database is too large (say someone storing their entire family photos, might have 5 or even 6 gigabytes of pictures). I don't think that this would run well (if at all) from memory. That's why I'm asking for a way to have the database just load necessary information (maybe table key columns for searching and indexing and then only the data requested from the database?). This way I am able to keep everything in one database file and can retrieve the data without issue, regardless of the size or number of pictures stored.

@moosedude: Can you use username/passwords with Datasets? Are there any articles on this? If I'm going this route do you recommend I use an access database instead of just creating a typed dataset and saving it in binary form?

Thanks again guys.

If I'm still being unclear on my reasons or my needs please don't give up on me and ask me to clarify,
~FenixRoA

Share this post


Link to post
Share on other sites
Quote:

If my database is too large (say someone storing their entire family photos, might have 5 or even 6 gigabytes of pictures). I don't think that this would run well (if at all) from memory. That's why I'm asking for a way to have the database just load necessary information (maybe table key columns for searching and indexing and then only the data requested from the database?). This way I am able to keep everything in one database file and can retrieve the data without issue, regardless of the size or number of pictures stored.


I think you're worrying a little too much about this, work with the database as if it was a real-time system, retrieve the data direct from the database as and when it is required by your application. Try this first and don't worry too much about performance at this early stage.

Databases are designed to manage huge amounts of data, it is ok for you to use a database in this way without resorting to loading datasets into memory. The amount of "stuff" stored in a database shouldn't have such a great impact on the speed of data retrieval.

Share this post


Link to post
Share on other sites
configuring user security is specific to the database you're using, i.e. SQL Server, mySQL etc..

It cannot be configured and attached to a dataset, but it is specific to opening a database connection i.e. the connection string when you call SqlConnection(string); (although, I'm not sure how plausible this is when connecting to a access database)

I guess in your application, when you first run it, the person would be prompted for a user name and password, which can then be used to determine which username your code will use to retrieve data from the database.

Share this post


Link to post
Share on other sites
Moosedude, firstly I appreciate the support. But I don't see how I can just ignore this issue, as it will no doubt play an important role in how I design the application.

The reason I am encrypting the files in the database is to avoid ANYONE being able to access it (even techies that know how to read database files with their program of choice). However even this is not an issue.

The issue here is how to access the database without dragging out the entire file into memory each time. Is this not possible with the current framework? If not please say so. Please keep in mind that this application is not always on or necessarily turns on with the computer, so each time the application is loaded (say someone opens a file, closes the application then opens another file) I would need to save and then reload the database.

I understand you shrugging it off as unimportant but loading even a database as small as a few hundred megabytes and saving it back every few seconds will discourage people from using my application.

I think what I may resort to doing (as I have yet to hear any possible schemes for using the database directly) is make a 'blob' file, and just have the database point to where the picture is in this file, rather then actually saving the picture inside the database.

If anyone can pose a better solution please post it here. Thanks again for your continued help,
~Barak

Share this post


Link to post
Share on other sites
Quote:
Original post by FenixRoA
The reason I am encrypting the files in the database is to avoid ANYONE being able to access it (even techies that know how to read database files with their program of choice). However even this is not an issue.


Depending on the db, the permission/role system will give you strong protection (as strong as the underlying os authentication). more powerful db's should run in their own account which will prevent snooping a raw dump of the db file if anything was possible to be scavanged. this is a basic problem that a db's must address generally - think payroll/finance/privacy/web server backed db.

Quote:
The issue here is how to access the database without dragging out the entire file into memory each time. Is this not possible with the current framework?


If you store the image as a blob in a table, then you can selectively choose to :-
(1) only read the associated meta data (name of image, date stored, size etc)
(2) read the actual image data or
(3) read both at the same time

Quote:
If not please say so. Please keep in mind that this application is not always on or necessarily turns on with the computer, so each time the application is loaded (say someone opens a file, closes the application then opens another file) I would need to save and then reload the database.


If a db gets started and stopped there's generally very little to do, spawn a listening connection for clients maybe. db's certainly dont load the data they serve into memory when they are started. (cf forcing the db to use memory store - but its not relevant here)

Quote:
I understand you shrugging it off as unimportant but loading even a database as small as a few hundred megabytes and saving it back every few seconds will discourage people from using my application.


Realational db is like a glorified filesystem with added features and better guarantees. when the os boots it doesnt load the entire filesystem into memory - just a few hooks. same with db. you are not going to find a system (db or not) that will scale up from reading and write a few hundred megabyes from disk every few seconds (clustered solution).

Quote:
I think what I may resort to doing (as I have yet to hear any possible schemes for using the database directly) is make a 'blob' file, and just have the database point to where the picture is in this file, rather then actually saving the picture inside the database.


Lots of people recommend doing this with raster type images in different contexts - ie mapserver backed by a GIS db. I tend to disagree with this since you loose so many of the advantages of a db and create issues /replication is much more complicated /backup / synchronisation. my view is that its better to seperate the binary image and the metadata into 2 tables for performance /normalisation and create a view join to compose the interface for clients - this is the basic application of a relation.

You should experiment with a real rdb - follow an sql tutorial and just work out what it will and will not do and how it could be adapted to your specific need - its both interesting and good general knowledge to have.

For c# i think ADO.NET will allow connections with access databases (using jet driver) or sql express. alternatively mysql and the much simpler file based sqlite are good alternatives (sqlite maybe only c++ ).

<edit clarity>

[Edited by - chairthrower on July 25, 2008 4:01:38 AM]

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!