Sign in to follow this  

Database suiting my needs?

This topic is 4730 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey guys, I'm working on a application which need to work with a lot of data, it seems that there is now too much data to deal with so my best choice would be to user a database. (Instead of storing all the data in memory).. I don't want a client/server database, I want everything to be in a stand-alone application. I heard about SQLite ( www.sqlite.org ) and it looks great, perfect for what I need. Unfortunatly, after many hours wasted, I could never get it to work with Borland C++ Builder. I first of all had a lot of difficulties getting it to link properly, once it finally compiled and linked with no problem, SQLite just makes my app crashes at startup. So, I think I will look for another solution, there must be something else which would work and fits all my need. I'd prefer a SQL driven database, but if you know a non-SQL database which would fit my need, I'll still be interested. I need a database which will be small, simple and allow to save/open a whole database in a single file on the hard drive. Any suggestion? Thanks a lot.

Share this post


Link to post
Share on other sites
I would second retrying with SQLite, it is really the best embedded SQL engine out there (unless you have some very special needs). In truth it's not hard to write the bindings yourself (instead of trying to mess with a lib that's probably not compatible with Builder, as is often the case), at least as of the 2.x version.

Share this post


Link to post
Share on other sites
I think you can also give XML a try. I do not know of the features of TinyXML - if it will be helpfor to you, but I do know that XML is a good alternative solution to trying a database. I think that XML is a possible solution to any data problems - the trick is being able to get it to do exactly what you want - which in turn takes more time but can be worth it in the end. Just an idea.

- Drew

Share this post


Link to post
Share on other sites
Quote:
Original post by Drew_Benton
... but I do know that XML is a good alternative solution to trying a database.

That depends entirely on the type of data you want to store. Some types of data are better stored in XML, which provides a hierarchical and flexible structure, can be edited by hand, and can be easily manipulated by external tools. Other types of data are much better stored in a relational database like SQLite, which provides efficient access and manipulation of large volumes of data that follows a very rigid and pre-determined structure.

John B

Share this post


Link to post
Share on other sites
I have no experience with XML, and learning it would take too much time as I am limited in time for this project.

Well, I guess I'll try SQLite again, if you guys help me, we might get it to run.


Let's start with the beginning. I downloaded under the "Precompiled Binaries For Windows" the "sqlitedll-3_0_8.zip"
It contained the file "sqlite3.dll" and "sqlite3.def"..

I have some experience with DLLs, but I must say that I never used a .def file and I have no idea how does it work.
I know that it has something to do with the name of the calls and it is related to the linker, for a library/dll, but I still don't know what is the reason it is used and what exactly can I do with it?

I also downloaded the source files ( "sqlite-source-3_0_8.zip" ).

So well, I then added some basic SQLite calls: sqlite3_open(..) and sqlite3_close(..)
and then compiled. Of course, I got a linking error, "Unresolved external: _sqlite3_open" and "Unresolved external: _sqlite3_close".
This was quite normal, since I did not link the library.
So.. hum.. no library was provided, I didn't know what to do, untill I found a message on a forum, saying that I should use the programs IMPDEF and IMPLIB in Borland.
IMPDEF allow me to create a .def file from my DLL ( sqlite3.dll ) and IMPLIB allows me to create a .lib from my DLL too.

So, yay, I got a .lib with that, I added it to my project and it compiled fine!

But, as soon as I try to run my application: Yourprog.exe has encountered an error and needs to close.

In my debugger, I see a write error at 0x00000000 and in the stack, I see that it is from within sqlite3.dll.


I tried to compile the DLL myself, tried a lot of different things with the .def file, but it always ended up in the application crashing at run time.

Any idea?


Quote:
Original post by Michalson
I would second retrying with SQLite, it is really the best embedded SQL engine out there (unless you have some very special needs). In truth it's not hard to write the bindings yourself (instead of trying to mess with a lib that's probably not compatible with Builder, as is often the case), at least as of the 2.x version.


I'm not sure to understand what you mean by writing my own bindings?

Share this post


Link to post
Share on other sites
Ok so I was curious about SQLLite myself so I decided to give it a go. I too had problems trying to get it to work. However - try this link. It generates the .lib once you rebuild the source or this link for a whole list of options to compile with. Now go to this page to build a quick sample app. Now to make a database - look at this link I found. Before you do that though, you must call sqlite with a parameter of the database you want to make: "sqlite test.db". Then enter in the commands. When you exit the file will be done. Now copy that file into the new project and run! Here is the sample I modified to work with the last tutorial:


#include <stdlib.h>
#include <stdio.h>
#include "sqlite3.h"

#pragma comment (lib,"sql lite.lib")

static int callback(void *NotUsed, int argc, char **argv, char **azColName){
int i;
for(i=0; i<argc; i++){
printf("%s = %s\n", azColName[i], argv[i] ? argv[i] : "NULL");
}
printf("\n");
return 0;
}

int main(int argc, char **argv){
sqlite3 *db;
char *zErrMsg = 0;
int rc;

/*if( argc!=3 ){
fprintf(stderr, "Usage: %s DATABASE SQL-STATEMENT\n", argv[0]);
exit(1);
}*/

rc = sqlite3_open("test.db", &db);

if( rc )
{
fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));
sqlite3_close(db);
exit(1);
}
rc = sqlite3_exec(db, "select * from tbl1", callback, 0, &zErrMsg);
if( rc!=SQLITE_OK ){
fprintf(stderr, "SQL error: %s\n", zErrMsg);
}
sqlite3_close(db);
return 0;
}



I hope this helps some!

- Drew

Share this post


Link to post
Share on other sites
Thanks for the all the info Drew_Benton, unfortunatly, I'm using Borland C++ Builder and I definitly can't switch, so your informations don't apply for BCB :(

There's a page explaining how to build for Borland C++ Builder, unfortunatly, my application still crashes with a DLL and .lib compiled by myself.


BTW, when following the exact steps given in the Borland How To Compile documentation file, I have some warnings with the .DEF file:
The .DEF they tell me to use looks like that:

LIBRARY sqlite3.dll
EXPORTS
sqlite_open = _sqlite_open
sqlite_close = _sqlite_close
sqlite_exec = _sqlite_exec
sqlite_last_insert_rowid = _sqlite_last_insert_rowid
sqlite_error_string = _sqlite_error_string
sqlite_interrupt = _sqlite_interrupt
sqlite_complete = _sqlite_complete
sqlite_busy_handler = _sqlite_busy_handler
sqlite_busy_timeout = _sqlite_busy_timeout
sqlite_get_table = _sqlite_get_table
..etc



While the original .DEF, which came in the precompiled binaries for win32 download looked like that:

EXPORTS
_sqlite3_aggregate_context
_sqlite3_aggregate_count
_sqlite3_bind_blob
_sqlite3_bind_double
_sqlite3_bind_int
_sqlite3_bind_int64
_sqlite3_bind_null
_sqlite3_bind_parameter_count
_sqlite3_bind_parameter_index
_sqlite3_bind_parameter_name
..etc



And when using the .def suggered in the documentation, I get one of those warning for each line of the .def:
"Attempt to export non-public symbol: '_sqlite_xxxx'"

But it compiles fine with the "original" .def file.

Share this post


Link to post
Share on other sites
Thanks a lot, I'm gonna take a look at it.

Though, there's something I just noticed.. The BCB debugger wasn't working right, I thought the only fact of linking the SQLite functions made my application crash at execution time, but I was wrong, it only crashes at the moment where one of these functions is called.

So, I guess there must be something wrong with the linking.. The functions calls seem to not be linked properly to work with the actual functions in the DLL?

Any idea what am I doing wrong that could cause that?
During this time, I'm gonna take a look at SQLite implementation for Borland from Aducom Software..

Share this post


Link to post
Share on other sites
Btw, I'm still open to suggestions for other ways to store my data..

Basically, what I want to do is that:
My program will add strings, often, not too long strings, never longer than about 30 caracters.
Those strings must be stored on the hard drive since there can be many strings, taking a lot of memory ( too much to be all kept in the RAM ).
I need a way to verify if a string already exists before adding a new one.
Since there can be a lot of strings ( over 10 000 ), I need an efficient strings search system.

It's quite simple, I don't need anything fancy. Any suggestion?
Thanks again.

Share this post


Link to post
Share on other sites
Quote:
Original post by Hedos
Btw, I'm still open to suggestions for other ways to store my data..

Basically, what I want to do is that:
My program will add strings, often, not too long strings, never longer than about 30 caracters.
Those strings must be stored on the hard drive since there can be many strings, taking a lot of memory ( too much to be all kept in the RAM ).
I need a way to verify if a string already exists before adding a new one.
Since there can be a lot of strings ( over 10 000 ), I need an efficient strings search system.

It's quite simple, I don't need anything fancy. Any suggestion?
Thanks again.


Just a note, 30,000, 30-character, 8-bit strings only take about 7.2MB of memory, which for most modern computers is an insignificant amount. You actually could just use a hashtable in memory because if your computer can't handle 7.2MB of data in memory then it shouldn't be used for modern applications anyways :)

Share this post


Link to post
Share on other sites
An hash table.. that could work.. I'll try that, thanks.
I'm just not sure if it will be easy to implement since it must be done from within a file?..

Share this post


Link to post
Share on other sites
Quote:
Original post by Hedos
Btw, I'm still open to suggestions for other ways to store my data..

Basically, what I want to do is that:
My program will add strings, often, not too long strings, never longer than about 30 caracters.
Those strings must be stored on the hard drive since there can be many strings, taking a lot of memory ( too much to be all kept in the RAM ).
I need a way to verify if a string already exists before adding a new one.
Since there can be a lot of strings ( over 10 000 ), I need an efficient strings search system.

It's quite simple, I don't need anything fancy. Any suggestion?
Thanks again.


jperalta's post gave me a quick idea - it may work for you. If you were to make a matrix that had at least 26 rows, one for each beginning letter + symbols, you could add strings based on the first character to the matrix. Then, you could make a binary search system for fast searching.

Here's a working example I made just for you (I too was in need of a database anyways) [smile] :

#include <vector>
#include <string>
using namespace std;

vector < vector< string > > Table;

void AddString( string str )
{
/* Variables we will use */
int insert_row = 0;
int tmp = str.at(0);

/* Determine where to insert at */
if( tmp >= 'a' && tmp <= 'z' )
insert_row = ( tmp - 97 );
else if( tmp >= 'A' && tmp <= 'Z' )
insert_row = ( tmp - 65 ) + 26;
else if( tmp >= '0' && tmp <= '9' )
insert_row = ( tmp - 48 ) + 52;
else
insert_row = 61;

/* Now check to see if it exists */
for( std::vector<string>::iterator itr = Table.at(insert_row).begin(); itr != Table.at(insert_row).end(); itr++ )
{
if( strcmp( (*itr).c_str(), str.c_str() ) == 0 )
return;
}

/* If we get here then we can add it */
Table.at(insert_row).push_back( str );
}

int main(int argc, char* argv[])
{
/* Set the initial size */
Table.resize( 62 ); // a-z, A-Z, 0-9, all other

/* Resize everything to empty */
for( int x = 0; x < 62; x++ )
{
Table.at(x).resize(0);
}

/* This proves that there are no string :) */
int size = 0;
for( x = 0; x < 62; x++ )
{
size += Table.at(x).size();
}
printf("%i\n",size);

/* Added some strings */
AddString("Drew");
AddString("Benton");
AddString("GameDev");

/* This proves that we added the strings :) */
size = 0;
for( x = 0; x < 62; x++ )
{
size += Table.at(x).size();
}
printf("%i\n",size);

/* Return */
return 0;
}



That is just the framework to work with. What you will need to customize the way it works - make it more efficient, but I would first try it as is with a large amount of strings to see if you really do need to make it store on the hard drive. I agree with jperalta that 7.2 MB is fine - windows has Virtual Memory it will use if the physical memory is full. I know it will work because I once loaded a 500+MB XML file I was testing out [wink]. I have a lot of system memory, but Windows will take care of those that don't. I hope this helps some!

- Drew

Share this post


Link to post
Share on other sites
Wow, thanks for that, I'll definitly take a look at that..
But, I have already implemented my hash table anyway, so I guess I'll keep it.
Still working on a minor bug, then I'll test with a lot of data, but so far it seems to work really good!

Share this post


Link to post
Share on other sites

This topic is 4730 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this