# Database suiting my needs?

This topic is 4730 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hey guys, I'm working on a application which need to work with a lot of data, it seems that there is now too much data to deal with so my best choice would be to user a database. (Instead of storing all the data in memory).. I don't want a client/server database, I want everything to be in a stand-alone application. I heard about SQLite ( www.sqlite.org ) and it looks great, perfect for what I need. Unfortunatly, after many hours wasted, I could never get it to work with Borland C++ Builder. I first of all had a lot of difficulties getting it to link properly, once it finally compiled and linked with no problem, SQLite just makes my app crashes at startup. So, I think I will look for another solution, there must be something else which would work and fits all my need. I'd prefer a SQL driven database, but if you know a non-SQL database which would fit my need, I'll still be interested. I need a database which will be small, simple and allow to save/open a whole database in a single file on the hard drive. Any suggestion? Thanks a lot.

##### Share on other sites
I'd probably spend some more time to fix sql lite if I were you.

##### Share on other sites
I would second retrying with SQLite, it is really the best embedded SQL engine out there (unless you have some very special needs). In truth it's not hard to write the bindings yourself (instead of trying to mess with a lib that's probably not compatible with Builder, as is often the case), at least as of the 2.x version.

##### Share on other sites
I think you can also give XML a try. I do not know of the features of TinyXML - if it will be helpfor to you, but I do know that XML is a good alternative solution to trying a database. I think that XML is a possible solution to any data problems - the trick is being able to get it to do exactly what you want - which in turn takes more time but can be worth it in the end. Just an idea.

- Drew

##### Share on other sites
Quote:
 Original post by Drew_Benton... but I do know that XML is a good alternative solution to trying a database.

That depends entirely on the type of data you want to store. Some types of data are better stored in XML, which provides a hierarchical and flexible structure, can be edited by hand, and can be easily manipulated by external tools. Other types of data are much better stored in a relational database like SQLite, which provides efficient access and manipulation of large volumes of data that follows a very rigid and pre-determined structure.

John B

##### Share on other sites
I have no experience with XML, and learning it would take too much time as I am limited in time for this project.

Well, I guess I'll try SQLite again, if you guys help me, we might get it to run.

It contained the file "sqlite3.dll" and "sqlite3.def"..

I have some experience with DLLs, but I must say that I never used a .def file and I have no idea how does it work.
I know that it has something to do with the name of the calls and it is related to the linker, for a library/dll, but I still don't know what is the reason it is used and what exactly can I do with it?

So well, I then added some basic SQLite calls: sqlite3_open(..) and sqlite3_close(..)
and then compiled. Of course, I got a linking error, "Unresolved external: _sqlite3_open" and "Unresolved external: _sqlite3_close".
This was quite normal, since I did not link the library.
So.. hum.. no library was provided, I didn't know what to do, untill I found a message on a forum, saying that I should use the programs IMPDEF and IMPLIB in Borland.
IMPDEF allow me to create a .def file from my DLL ( sqlite3.dll ) and IMPLIB allows me to create a .lib from my DLL too.

So, yay, I got a .lib with that, I added it to my project and it compiled fine!

But, as soon as I try to run my application: Yourprog.exe has encountered an error and needs to close.

In my debugger, I see a write error at 0x00000000 and in the stack, I see that it is from within sqlite3.dll.

I tried to compile the DLL myself, tried a lot of different things with the .def file, but it always ended up in the application crashing at run time.

Any idea?

Quote:
 Original post by MichalsonI would second retrying with SQLite, it is really the best embedded SQL engine out there (unless you have some very special needs). In truth it's not hard to write the bindings yourself (instead of trying to mess with a lib that's probably not compatible with Builder, as is often the case), at least as of the 2.x version.

I'm not sure to understand what you mean by writing my own bindings?

##### Share on other sites
Ok so I was curious about SQLLite myself so I decided to give it a go. I too had problems trying to get it to work. However - try this link. It generates the .lib once you rebuild the source or this link for a whole list of options to compile with. Now go to this page to build a quick sample app. Now to make a database - look at this link I found. Before you do that though, you must call sqlite with a parameter of the database you want to make: "sqlite test.db". Then enter in the commands. When you exit the file will be done. Now copy that file into the new project and run! Here is the sample I modified to work with the last tutorial:

#include <stdlib.h>#include <stdio.h>#include "sqlite3.h"#pragma comment (lib,"sql lite.lib")static int callback(void *NotUsed, int argc, char **argv, char **azColName){  int i;  for(i=0; i<argc; i++){    printf("%s = %s\n", azColName[i], argv[i] ? argv[i] : "NULL");  }  printf("\n");  return 0;}int main(int argc, char **argv){  sqlite3 *db;  char *zErrMsg = 0;  int rc;  /*if( argc!=3 ){    fprintf(stderr, "Usage: %s DATABASE SQL-STATEMENT\n", argv[0]);    exit(1);  }*/  rc = sqlite3_open("test.db", &db);     if( rc )  {    fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));    sqlite3_close(db);    exit(1);  }  rc = sqlite3_exec(db, "select * from tbl1", callback, 0, &zErrMsg);  if( rc!=SQLITE_OK ){    fprintf(stderr, "SQL error: %s\n", zErrMsg);  }  sqlite3_close(db);  return 0;}

I hope this helps some!

- Drew

##### Share on other sites
Thanks for the all the info Drew_Benton, unfortunatly, I'm using Borland C++ Builder and I definitly can't switch, so your informations don't apply for BCB :(

There's a page explaining how to build for Borland C++ Builder, unfortunatly, my application still crashes with a DLL and .lib compiled by myself.

BTW, when following the exact steps given in the Borland How To Compile documentation file, I have some warnings with the .DEF file:
The .DEF they tell me to use looks like that:
   LIBRARY sqlite3.dll   EXPORTS   sqlite_open    = _sqlite_open   sqlite_close   = _sqlite_close   sqlite_exec    = _sqlite_exec   sqlite_last_insert_rowid   = _sqlite_last_insert_rowid   sqlite_error_string   = _sqlite_error_string   sqlite_interrupt   = _sqlite_interrupt   sqlite_complete   = _sqlite_complete   sqlite_busy_handler   = _sqlite_busy_handler   sqlite_busy_timeout   = _sqlite_busy_timeout   sqlite_get_table   = _sqlite_get_table   ..etc

While the original .DEF, which came in the precompiled binaries for win32 download looked like that:
EXPORTS_sqlite3_aggregate_context_sqlite3_aggregate_count_sqlite3_bind_blob_sqlite3_bind_double_sqlite3_bind_int_sqlite3_bind_int64_sqlite3_bind_null_sqlite3_bind_parameter_count_sqlite3_bind_parameter_index_sqlite3_bind_parameter_name..etc

And when using the .def suggered in the documentation, I get one of those warning for each line of the .def:
"Attempt to export non-public symbol: '_sqlite_xxxx'"

But it compiles fine with the "original" .def file.

##### Share on other sites
Take a look at this site. I have used Builder before so I know how hard it can be at times to add 3rd party components. It is about the only thing I can find about using SQLite with Builder. I hope it can be of some use. If not good luck!

- Drew

##### Share on other sites
Thanks a lot, I'm gonna take a look at it.

Though, there's something I just noticed.. The BCB debugger wasn't working right, I thought the only fact of linking the SQLite functions made my application crash at execution time, but I was wrong, it only crashes at the moment where one of these functions is called.

So, I guess there must be something wrong with the linking.. The functions calls seem to not be linked properly to work with the actual functions in the DLL?

Any idea what am I doing wrong that could cause that?
During this time, I'm gonna take a look at SQLite implementation for Borland from Aducom Software..

##### Share on other sites
Btw, I'm still open to suggestions for other ways to store my data..

Basically, what I want to do is that:
My program will add strings, often, not too long strings, never longer than about 30 caracters.
Those strings must be stored on the hard drive since there can be many strings, taking a lot of memory ( too much to be all kept in the RAM ).
I need a way to verify if a string already exists before adding a new one.
Since there can be a lot of strings ( over 10 000 ), I need an efficient strings search system.

It's quite simple, I don't need anything fancy. Any suggestion?
Thanks again.

##### Share on other sites
Quote:
 Original post by HedosBtw, I'm still open to suggestions for other ways to store my data..Basically, what I want to do is that:My program will add strings, often, not too long strings, never longer than about 30 caracters.Those strings must be stored on the hard drive since there can be many strings, taking a lot of memory ( too much to be all kept in the RAM ).I need a way to verify if a string already exists before adding a new one.Since there can be a lot of strings ( over 10 000 ), I need an efficient strings search system.It's quite simple, I don't need anything fancy. Any suggestion?Thanks again.

Just a note, 30,000, 30-character, 8-bit strings only take about 7.2MB of memory, which for most modern computers is an insignificant amount. You actually could just use a hashtable in memory because if your computer can't handle 7.2MB of data in memory then it shouldn't be used for modern applications anyways :)

##### Share on other sites
An hash table.. that could work.. I'll try that, thanks.
I'm just not sure if it will be easy to implement since it must be done from within a file?..

##### Share on other sites
Quote:
 Original post by HedosBtw, I'm still open to suggestions for other ways to store my data..Basically, what I want to do is that:My program will add strings, often, not too long strings, never longer than about 30 caracters.Those strings must be stored on the hard drive since there can be many strings, taking a lot of memory ( too much to be all kept in the RAM ).I need a way to verify if a string already exists before adding a new one.Since there can be a lot of strings ( over 10 000 ), I need an efficient strings search system.It's quite simple, I don't need anything fancy. Any suggestion?Thanks again.

jperalta's post gave me a quick idea - it may work for you. If you were to make a matrix that had at least 26 rows, one for each beginning letter + symbols, you could add strings based on the first character to the matrix. Then, you could make a binary search system for fast searching.

Here's a working example I made just for you (I too was in need of a database anyways) [smile] :
#include <vector>#include <string>using namespace std;vector < vector< string > > Table;void AddString( string str ){	/*  Variables we will use  */	int insert_row = 0;	int tmp = str.at(0);	/*  Determine where to insert at  */	if( tmp >= 'a' && tmp <= 'z' )		insert_row = ( tmp - 97 );	else if( tmp >= 'A' && tmp <= 'Z' )		insert_row = ( tmp - 65 ) + 26;	else if( tmp >= '0' && tmp <= '9' )		insert_row = ( tmp - 48 ) + 52;	else		insert_row = 61;	/*  Now check to see if it exists  */	for( std::vector<string>::iterator itr = Table.at(insert_row).begin(); itr != Table.at(insert_row).end(); itr++ )	{		if( strcmp( (*itr).c_str(), str.c_str() ) == 0 )			return;	}	/*  If we get here then we can add it  */	Table.at(insert_row).push_back( str );}int main(int argc, char* argv[]){	/*  Set the initial size  */	Table.resize( 62 );	// a-z, A-Z, 0-9, all other		/*  Resize everything to empty  */	for( int x = 0; x < 62; x++ )	{		Table.at(x).resize(0);	}	/*  This proves that there are no string :)  */	int size = 0;	for( x = 0; x < 62; x++ )	{		size += Table.at(x).size();	}	printf("%i\n",size);	/*  Added some strings  */	AddString("Drew");	AddString("Benton");	AddString("GameDev");	/*  This proves that we added the strings :)  */	size = 0;	for( x = 0; x < 62; x++ )	{		size += Table.at(x).size();	}	printf("%i\n",size);	/*  Return  */	return 0;}

That is just the framework to work with. What you will need to customize the way it works - make it more efficient, but I would first try it as is with a large amount of strings to see if you really do need to make it store on the hard drive. I agree with jperalta that 7.2 MB is fine - windows has Virtual Memory it will use if the physical memory is full. I know it will work because I once loaded a 500+MB XML file I was testing out [wink]. I have a lot of system memory, but Windows will take care of those that don't. I hope this helps some!

- Drew

##### Share on other sites
Wow, thanks for that, I'll definitly take a look at that..
But, I have already implemented my hash table anyway, so I guess I'll keep it.
Still working on a minor bug, then I'll test with a lot of data, but so far it seems to work really good!

##### Share on other sites

This topic is 4730 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Create an account

Register a new account

• ### Forum Statistics

• Total Topics
628710
• Total Posts
2984325

• 23
• 11
• 9
• 13
• 14