#include <fstream>#include <iostream>#include <string>#include <vector>#include <sstream>using std::cout;using std::endl;std::ifstream fin("histY.csv.txt");std::string buff, elem;std::vector<std::string> fields;// Split by LFwhile( std::getline( fin, buff, char(10) ) ){	std::stringstream ss;	ss << buff;	while( std::getline( ss, elem, ',') )	{		// Erase the leading space if one is there		if( elem[0] == ' ' )			elem.erase( elem.begin() );		// Erase the trailing space if one is there		if( elem[elem.size() - 1] == ' ' )			elem.erase( --elem.end() );		// Show the element		cout << elem << endl;                // Now do whatever you want with each element here, probabally add it all the the vector	}        // When you get here, you have a vector of fileds, now you send it to the DB how you were doing it.}fin.close();

first name,last name, address, city, state, month1, month2, month3first name,last name, address, city, state, month1, month2, month3

Quote:Original post by graveyard filla
Here is the function which takes the file and dumps it into the database. It worked fine with a 400 meg file (dumped 3.5 mill records). But I have another file that is 900 megs, and this is the one that gave me the "low on virtual memory" dealy. BTW, this code needs to be changed slightly to do an insert into 2 tables to be fully normalized.

This is for a power company.... this table has the power usage for like 8 million customers.. The problem is the schema they had was HORRIBLE, there table looked like this:

First Name, Last Name, Address, City, (etc), MonthUsage1, MonthUsage2, MonthUsage3, MonthUsage4, MonthUsage5.... etc. Not only did they cram 12 fields into the one table, but they made it RELATIVE, so MonthUsage1 isnt January, its actually August.

*** Source Snippet Removed ***

Quote:Original post by graveyard filla
Pragma, I'm trying to understand your example, but how does fscanf() know how many bytes / characters to read? And how does it know to ignore commas, etc.?

Quote:Original post by graveyard filla
It doesn't have to store data for more then 12 months

Quote:Original post by graveyard filla
Drew, from your example of getline(), it doesn't seem to fit with the reference i found for getline()... I'm guessing I should do something like fin.getline(&some_string,999999,"'") ? I put 999999 there because I want it to keep reading untill it finds the deliminator.

12  3   4   5 6

Quote:
wow, thats a lot nicer then my code . one thing that i don't get, whats with the 2 lines where you trim the whitespace off the ends of the field? why did you do that?