Jump to content
  • Advertisement
Sign in to follow this  
Snowthief

Development speed using databases

This topic is 4251 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi. I'm an old programmer who has stuck with C++ until now and have finally decided to move on, because I don't believe I'm doing things the optimal way. Let's say I have a struct containing some data. I need to get certain values here, could be some names sorted alphabetically, you know how I would do that? Create a database, insert my data, SELECT FROM, delete database. Now, I realize databases are very good for handling data, because that's obviously what they were built for, but I also realize that this just isn't right. There has to be a better way. I used to do it the slow way in C++, writing sorting algorithms and lots of for-nexts, but then I did some web development for a while which introduced me to databases and now I find myself using them for EVERYTHING, because it feels like so much code has already been written for me. A more complicated example: "SELECT TOP 3 DISTINCT a FROM MyTable WHERE b-c > 0 AND d = 10 GROUP BY e ORDER BY f DESC" That line takes me no more than 15 seconds to write. Do you know how long this would take me to code in C++? I don't even want to think about that. "UPDATE MyTable SET a = 0 WHERE b > c" One line, takes 5 seconds to write, makes for some extremely neat code that's very easy to maintain. If I put the same thing on one line in C++ to deal with data in a struct, it would be a very long line, not to mention that it would require a variable just to keep track of how far we are in the update process. So, which language should I switch to? Development speed is very important to me, code execution speed is not. I'm looking for a language that either has built-in features that make sorting easy or a library for a language that does. It doesn't have to be "text statements" like SQL has, it just has to make for neat code that's easy to maintain and most importantly is very fast to code. That means I don't want to waste my time coding and implementing sorting algorithms. I'm looking for something that's simple and clever. I don't want to have to write 20 lines of code to do something as simple as picking the 10 highest values of MyStruct[x].MyValue. Thank you for your time, I greatly appreciate it.

Share this post


Link to post
Share on other sites
Advertisement
I'm tempted to recommend OCaml or Haskell, because they will blow your mind (and expand it), but it really sounds like you really want Python. Reasonably elegant, reasonably performant, and lightning-fast to develop in. I don't write pseudocode anymore; I write Python.

Share this post


Link to post
Share on other sites
You might consider googling "functional programming" (it is natively supported by things like OCaml and Haskell, and also Lisp in a way; Python provides easy ways to do it although it isn't fundamentally an FP language; C++ supports the idiom via "advanced" stuff in the standard library).

There are certain things that really *are* better done by the database, though. OTOH, they kinda suck at graphics ;) Two major categories of WTFs on thedailywtf are (a) doing calculations in code that the database could do much more easily (typically this involves doing the "where" logic of a "select" in code; this is often much, much slower!) and (b) doing calculations in the database that the code could do much more easily (arcane and hideously complex stored procedures, for example).

Share this post


Link to post
Share on other sites
Quote:
I used to do it the slow way in C++

Why not do it the fast way in C++ and use the standard library? Of course your code is going to be bloated if you create everything from first principles - you shouldn't need to write a sorting algorithm in C++ - there's already std::sort.

That said, some other languages are more compact and expressive than C++:

Now, some examples:
Quote:
I'm looking for something that's simple and clever. I don't want to have to write 20 lines of code to do something as simple as picking the 10 highest values of MyStruct[x].MyValue


In Python:

#declare the necessary stuff
class MyStruct:
def __init__(self):
self.MyValue = 0

MyData = []#assume MyData is filled with MyStructs

#Now the algorithm:

#sort the data
MyData.sort( lambda left,right : -cmp(left.MyValue,right.MyValue) )

#return the ten highest values
retur MyData[:10]






Now in C++

//Declare the necessary stuff
#include <algorithm>
#include <vector>
#include <boost/bind.hpp>
using namespace boost;

struct MyStruct
{
int MyValue;
};
std::vector<MyStruct> MyData;

//Now the algorithm:

//sort the data
std::sort(MyData.begin(),MyData.end(),bind(&MyStruct::MyValue,_1) > bind(&MyStruct::MyValue,_2));

//return the ten highest values
return std::vector<MyStruct>(MyData.begin(),MyData.begin()+std::min<size_t>(MyData.size(),10));






As you can see, the algorithm itself is 2 lines in both languages. However, some of the stuff in the C++ example has a much higher learning curve than the Python, and I'm much more confident that the Python example is bug free than I am of the C++ one.

[Edited by - Nitage on January 31, 2007 12:32:21 PM]

Share this post


Link to post
Share on other sites
Yes, I think you'd appreciate Python. As of 2.5, you have lightweight SQL functionality built in via the SQLite library:


>>> import sqlite3
>>> c = conn.cursor()
>>> c.execute('select * from stocks order by price')
>>> for row in c:
... print row
...
(u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001)
(u'2006-03-28', u'BUY', u'IBM', 1000, 45.0)
(u'2006-04-06', u'SELL', u'IBM', 500, 53.0)
(u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0)



Or if you want most of the ACID properties of databases but not the SQL syntax, an object-relational mapper like SQLObject may suit you:


>>> from sqlobject import *
>>>
>>> sqlhub.processConnection = connectionForURI('sqlite:/:memory:')
>>>
>>> class Person(SQLObject):
... fname = StringCol()
... mi = StringCol(length=1, default=None)
... lname = StringCol()
...
>>> Person.createTable()
>>> p = Person(fname="John", lname="Doe")
>>> p
<Person 1 fname='John' mi=None lname='Doe'>
>>> p.fname
'John'
>>> p.mi = 'Q'
>>> p2 = Person.get(1)
>>> p2
<Person 1 fname='John' mi='Q' lname='Doe'>
>>> p is p2
True


Note that in this example uses an in-memory database but you only have to change the connectionForURI call to have it as a disk-based one.

Share this post


Link to post
Share on other sites
Quote:
Original post by Nitage
Quote:
I used to do it the slow way in C++

Why not do it the fast way in C++ and use the standard library? Of course your code is going to be bloated if you create everything from first principles - you shouldn't need to write a sorting algorithm in C++ - there's already std::sort.
I couldn't agree more.
You don't really write sorting algorithms in C++ unless the SC++L code isn't sufficient (which is practically never).

Quote:
... databases and now I find myself using them for EVERYTHING ...
This reminds me of a saying:
"Give a man a hammer and he will think all problems are nails."

Creating a database only to destroy it moments later, for example, is just absurd!

Share this post


Link to post
Share on other sites
and in the .NET world, there is LINQ for the next version of the languages.

http://msdn2.microsoft.com/en-us/netframework/aa904594.aspx

scroll a little down to the "videos" section and watch the 4 "... in action" videos (skim forward in the first few minutes of each additional video as they lead in somewhat similarly I think).

Share this post


Link to post
Share on other sites
Thank you for your comments and suggestions everyone, Python sure does look clever, but I'm not sure it's the right way to go for me.

"Why not do it the fast way in C++ and use the standard library?"

Your examples are good, but they still only do a very small part of what can be done with SQL. I understand this, of course, because like I said, so much code has already been written for you when you use SQL. My point is, that what if you wanted to output MyStruct[x].Name as well for the sorted values? What if you wanted to output only unique values? What if you wanted to group the results in a certain way? What if you wanted to link the results to another struct and compare with those? std::sort will only get you so far and there are so many possible scenarios that I could code so much faster using SQL.

"Give a man a hammer and he will think all problems are nails."

Well, I have been using the C++ screwdriver for many years, but then my boss forced me to use an SQL hammer for a while and then I switched to just hammering my screws in, because I realized that was much, much faster. :)

"Creating a database only to destroy it moments later, for example, is just absurd!"

I couldn't agree more, that's why I posted here. :)

"and in the .NET world, there is LINQ for the next version of the languages."

Bril-li-ant. This is EXACTLY what I'm looking for. I have almost no experience with C#, but C# + LINQ appears to be exactly what I'm looking for, so I'll go try that right now. :)

Once again, thank you for your comments and suggestions everyone, I greatly appreciate it. Perhaps one day I will be ready for Python, but I don't think I'm there yet, possibly because I've been developing in C/C++ for almost 20 years now, so it's hard to move on to something so different. :)

Thank you again for your time. :)

Share this post


Link to post
Share on other sites
It's my understanding that LINQ hasn't been released, so while it looks useful it isn't a solution yet (and would require you to switch to .NET - which might not be appropriate given that .NET is Windows only - and Mono obviously doesn't implement it yet). It looks really nice though.

Quote:

Your examples are good, but they still only do a very small part of what can be done with SQL. I understand this, of course, because like I said, so much code has already been written for you when you use SQL.


Ok, I'll have a go at the more complicated query you gave:
SELECT TOP 3 DISTINCT a FROM MyTable WHERE b-c > 0 AND d = 10 GROUP BY e ORDER BY f DESC

Translates into C++ roughly as:

//create a copy of the input, as we're don't want to corrupt the original data
std::vector<MyStruct> records(MyTable);

//Perform the 'WHERE' clause:
records.erase(records.begin(),std::remove_if(records.begin(),records.end(),(bind(MyStruct::a,_1)-bind(MyStruct::b,_1) > 0) && (bind(MyStruct::d,_1) == 10)));

//Remove all the duplicates based on value of 'a'
std::sort(records.begin(),records.end(),bind(MyStruct::a,_1) > bind(MyStruct::a,_2));
records.erase(records.begin(),std::unique(records.begin(),records.end(),bind(MyStruct::a,_1)==bind(MyStruct::a,_2)));

//Order the records according to f, and get only the top 3
std::sort(records.begin(),records.end(),bind(MyStruct::f,_1) > bind(MyStruct::f,_2));
records.assign(records.begin(),records.begin()+std::min(records.size(),3u));

//We want to select only the a values:
std::vector<int> results;
std::for_each(records.begin(),records.end(),bind(std::vector<int>::push_back,results,bind(MyStruct::a,_1));

return results;




So while it was fairly easy to write the simple query in C++, the more complicated one quickly gets out of hand. It's also (obviously) far less readable the the SQL.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!