🔒 Locked

Help with Unicode text files and std::ifstream

Started by Dookie Oct 16, 2008 at 3:23 PM 13 replies 9.5k views

Original Post

Oct 16, 2008 at 3:23 PM

Hello! I've searched for this issue before posting, but I can't find a solid answer to my issue. I have a program that reads in info from a text file using std::ifstream, but the text file may or may not be encoded in Unicode. I stumbled upon this fact when I saw two files with seemingly identical contents, but with two completely different file sizes... Open both files in Notepad, and they look completely identical, but they have two different file sizes. What's frustrating me is that one can be opened and read by 'std::ifstream', and one cannot. All I want to do is open a text file that may be encoded in Unicode. This doesn't work:

ifstream inTxt("inFile.txt");

Any suggestions? Thanks in advance for the help!

Spoonbender

Oct 16, 2008 at 4:06 PM

What do you mean, it "doesn't work"? std::ifstream doesn't care about the encoding, it just reads the file as a stream of char's (= bytes)

LanceRC

Oct 16, 2008 at 4:13 PM

if you want to open unicode text files, you'll need to look for the byte order mark and process accordingly.

Dookie

Oct 16, 2008 at 4:21 PM

Sorry I wasn't more specific about 'it doesn't work', I'm just frustrated with it and some of my hair is missing from pulling it out. D'oh!

When I open the file, it opens OK without errors. But when I read data from it using "inTxt.getline(txtStuff, 2048);", the variable 'txtStuff' contains garbage if it's a Unicode text file. The variable 'txtStuff' contains a line of readable text if the text file is a plain vanilla text file.

Other things are frustrating me today, but I digress. Sorry if I sound ticked off, I don't mean to come off that way. :(

reptor

Oct 16, 2008 at 5:01 PM

Have you tried

std::wifstream

?

there's also std::wstring

MaulingMonkey

Oct 16, 2008 at 5:35 PM

Due to unfortunate wording of the C++ language spec, std::wifstream also operates on bytes (meaning it just calls widen() for each byte read, which does nothing but change the type to wchar_t -- not actually interpret the file as UNICODE), so it's completely worthless to you as is. Unfortunately, for proper UNICODE support, you'll have to use something else (such as some of the functions found in the Win32 library) to read your files (unless you wish to implement UNICODE handling on top of the basic byte i/o yourself).

SiCrane

Oct 16, 2008 at 5:50 PM

You can get around some of the wfstream bizarre behavior by imbuing your stream with a null codecvt. Unfortunately that only does part of the work and it still won't be portable. (wchar_t is typically 16 bits on Windows systems but 32-bits on *nix systems, you've still got to deal with byte order, etc.) I personally recommend using ICU for Unicode processing though other internationalization libraries can be used instead.

Dookie

Oct 17, 2008 at 1:19 AM

Well, I'm starting to understand Unicode but it looks like portability of my code would be a pain in the poop chute. Good thing I don't care about portability... All I want is for this to work with a file saved in Windows Notepad (saves in UTF-8, I think):

void ScanThread(PVOID pvoid){	PPARAMS pparams;	// Scanner variables	char	startRes[2048];	char	endRes[2048];	int		strInfo;	ifstream	inStart, inEnd;	ofstream	outDiff;	pparams = (PPARAMS) pvoid;	while (!pparams->threadDone)	{		if (pparams->doCompare)		{			pparams->doCompare = false;			// open the file 'start.txt' for reading...			SetDlgItemText( pparams->hwnd, IDC_STAGENUM, "Stage 1" );			inStart.open("start.txt");			if (!inStart)			{				// No 'start' file...				// ------------------				pparams->retResult = 1;				goto EndCompare;			}			// and open the file 'end.txt' for reading...			SetDlgItemText( pparams->hwnd, IDC_STAGENUM, "Stage 2" );			inEnd.open("end.txt");			if (!inEnd)			{				// No 'end' file...				// ----------------				inStart.close();				pparams->retResult = 2;				goto EndCompare;			}			// create the file 'difference.txt' for writing...			SetDlgItemText( pparams->hwnd, IDC_STAGENUM, "Stage 3" );			outDiff.open("difference.txt");			if (!outDiff)			{				// Can't create 'difference' file...				// ---------------------------------				inStart.close();				inEnd.close();				pparams->retResult = 3;				goto EndCompare;			}			// First, compare every line in Start with End...			outDiff <<				"-----------------------------------------------------" << endl <<				"| Stuff found in Start that was different from End: |" << endl <<				"-----------------------------------------------------" << endl << endl;			SetDlgItemText( pparams->hwnd, IDC_STAGENUM, "Stage 4" );			while (!inStart.eof())			{				IncrementStartCounter(pparams->hwnd, pparams->StartNum);				inStart.getline(startRes, 2048);				inEnd.seekg(0, ios_base::beg);				inEnd.clear();				pparams->EndNum = 0;				while (!inEnd.eof())				{					IncrementEndCounter(pparams->hwnd, pparams->EndNum);					inEnd.getline(endRes, 2048);					strInfo = strcmp(startRes, endRes);					if (strInfo == 0)						goto NextIteration;				}				outDiff << startRes << endl;				NextIteration:				inEnd.clear();			}			inStart.seekg(0, ios_base::beg);			inStart.clear();			inEnd.seekg(0, ios_base::beg);			inEnd.clear();			// ...then compare every line in End with Start.			pparams->StartNum = 0;			pparams->EndNum = 0;			outDiff << endl << endl <<				"-----------------------------------------------------" << endl <<				"| Stuff found in End that was different from Start: |" << endl <<				"-----------------------------------------------------" << endl << endl;			SetDlgItemText( pparams->hwnd, IDC_STAGENUM, "Stage 5" );			while (!inEnd.eof())			{				IncrementStartCounter(pparams->hwnd, pparams->StartNum);				inEnd.getline(endRes, 2048);				inStart.seekg(0, ios_base::beg);				inStart.clear();				pparams->EndNum = 0;				while (!inStart.eof())				{					IncrementEndCounter(pparams->hwnd, pparams->EndNum);					inStart.getline(startRes, 2048);					strInfo = strcmp(startRes, endRes);					if (strInfo == 0)						goto NextIteration2;				}				outDiff << endRes << endl;				NextIteration2:				inStart.clear();			}			// We're done!			outDiff << endl << endl <<				"==============" << endl <<				"Scan complete!" << endl;			inStart.close();			inEnd.close();			outDiff.close();			SetDlgItemText( pparams->hwnd, IDC_STAGENUM, "Finished!" );			EndCompare:			pparams->compareDone = true;		}		Sleep(50);	}	_endthread();}

The above code is a comparison program thread that looks at two text files and then outputs any differences to a third file, and it works perfectly as long as the 'Start.txt' and 'End.txt' files are plain vanilla text files. If I try to run this routine with the same text files after saving them in Notepad (Unicode 8, I think), then it outputs a couple of control characters and nothing else.

How in tarnation do I get the above code to work with Unicode-encrusted Windows Notepad text files? I just can't figure it out. [Mr Bump] Poopity-poop! [/Mr Bump]

Thanks in advance for the help!

SiCrane

Oct 17, 2008 at 8:37 AM

Windows Notepad will save Unicode files in one of three formats: UTF-8, UTF-16LE and UTF-16BE. In all three formats the file will be prepended with a byte order marker (BOM). For a UTF-8 file this means the first three bytes will be EF BB BF. For UTF-16LE, the first two bytes will be FF FE, and for UTF-16BE, the first two bytes will be FE FF. You'll need to decide how you want to handle files without a BOM; generally you can assume an ASCII encoding, which being a subset of UTF-8 can usually be processed as if they are UTF-8. Once you decide on how your files are encoded you need to read the files and convert them to a common internal encoding in order to do any comparisons.

With ICU you can use ucnv_detectUnicodeSignature() to get the encoding from a BOM. You can use ucnv_open() to take that encoding and create a converter to load data in that encoding. ucnv_toUnicode() can be used with that converter to convert text in the file to a internal Unicode encoding. From there you can do whatever internal processing you want.

Zahlman

Oct 17, 2008 at 1:24 PM

.... Wow. Threading sure gets ugly fast, hmm?

Some proposals:

void setStage(HWND hwnd, int stage) {	SetDlgItemText(hwnd, IDC_STAGENUM, (std::stringstream("Stage ") << stage).str().c_str());}void reset(std::ifstream& stream) {	stream.seekg(0, ios_base::beg);	stream.clear();}void compareFiles(HWND hwnd, std::ifstream& a, int& acounter, std::ifstream& b, int& bcounter, std::ofstream& out) {	std::string aLine, bLine;	while (std::getline(a, aLine)) {		IncrementStartCounter(pparams->hwnd, acounter);		reset(b);		bcounter = 0;		bool found = false;		while (std::getline(b, bLine)) {			IncrementEndCounter(pparams->hwnd, bcounter);			if (aLine == bLine) { found = true; break; }		}		if (!found) { out << aLine << endl; }	}}int ScanThread_helper(PPARAMS pparams) {	setStage(pparams->hwnd, 1);	ifstream inStart("start.txt");	if (!inStart) {		return 1;	}	setStage(pparams->hwnd, 2);	ifstream inEnd("end.txt");	if (!inEnd) {		return 2;	}	setStage(pparams->hwnd, 3);	ofstream outDiff("difference.txt");	if (!outDiff) {		return 3;	}	outDiff <<		"-----------------------------------------------------\n"		"| Stuff found in Start that was different from End: |\n"		"-----------------------------------------------------\n" << endl;	setStage(pparams->hwnd, 4);	compareFiles(pparams->hwnd, inStart, pparams->StartNum, inEnd, pparams->EndNum, outDiff);	reset(inStart);	reset(inEnd);	pparams->StartNum = 0;	pparams->EndNum = 0;				outDiff << endl << endl <<		"-----------------------------------------------------\n"		"| Stuff found in End that was different from Start: |\n"		"-----------------------------------------------------\n" << endl;	setStage(pparams->hwnd, 5);	compareFiles(pparams->hwnd, inEnd, pparams->EndNum, inStart, pparams->StartNum, outDiff);	// We're done!	outDiff << "\n\n==============\nScan complete!" << endl;	SetDlgItemText(pparams->hwnd, IDC_STAGENUM, "Finished!");	return 0;}void ScanThread(PVOID pvoid) {	PPARAMS pparams = reinterpret_cast<PPARAMS*>(pvoid);	while (!pparams->threadDone) {		if (pparams->doCompare) {			pparams->doCompare = false;			pparams->retResult = ScanThread_helper(pparams);			pparams->compareDone = true;		}		Sleep(50);	}	_endthread();}

Dookie

Oct 17, 2008 at 4:26 PM

I tried your idea of going with ICU SiCrane, but I'm drowning in 'Can't find xxx.lib' and 'A tool returned an error code: "Performing Custom Build Step"' and similar errors to the point where I'm ripping out even more hair than before. I have VisualC 7.0, and the code/binaries at ICU are either 6.0 or 7.1 or 8.0. On top of that, the souce code won't compile because it's looking for libraries that weren't packaged with the code (icuuc.lib, icuucd.lib, etc)... Must be one of those things where Windows programming environments are an afterthought, being as how 'Microsoft' and 'Windows' aren't even mentioned on their home page.

Any other ideas, or can this even be done in my programming environment of choice? Or rather, the only programming environment I know?

Thanks for cleaning up my code, Zahlman! I just threw it together as I was experimenting with different ideas, so it got sloppy fast.

Thanks!

SiCrane

Oct 17, 2008 at 5:11 PM

Given that no operating system is listed on the ICU homepage, complaining about a lack of "Microsoft" or "Windows" being listed is rather bizarre. If you look at the list of reference platforms you'll see that Windows Vista SP1 and Windows Server 2003 are both on that list for ICU 4.0. Meaning that both those operating systems are ones that they develop the library on.

If you download a binary package for ICU on Windows, you'll find that the libs you've mentioned as missing are actually in the ICU/libs folder. Older versions of MSVC should be able to link with newer versions of the library, but you may need to install the redistributable for the compiler it was compiled with. However, this may be a good time to upgrade to a newer version. MSVC 2008 Express Edition is available for download from Microsoft for free. If you insist on using an outdated compiler, then the last ICU package I know of that was tested with MSVC 7.0 was 3.4.1; you should at least be able to get that to work with your compiler.

Each of the ICU versions has an associated "ReadMe" file. For example, for ICU 4.0, the ReadMe file is here and a copy comes with the source packages. These give detailed instructions of how to build and install ICU for a variety of platforms. You'll find that the Windows instructions are actually at the top of the platform specific directions.

Dookie

Oct 17, 2008 at 6:41 PM

Thanks for the info, SiCrane. I'll check out that readme file and see if I can figure it out... By the way, how good is that Visual Express Edition? Will it work fine with Windows XP, or is it more designed with Vista in mind? It might be time for me to upgrade to a new programming environment.

Spoonbender

Oct 17, 2008 at 6:45 PM

Quote:
Original post by Dookie
Thanks for the info, SiCrane. I'll check out that readme file and see if I can figure it out... By the way, how good is that Visual Express Edition? Will it work fine with Windows XP, or is it more designed with Vista in mind? It might be time for me to upgrade to a new programming environment.

Works fine with either OS. (And I'm pretty sure I read on the VC++ team's blog that the next version of Visual Studio will also run on XP, so no need to worry about upgrading OS any time soon)

MaulingMonkey

Oct 17, 2008 at 8:41 PM

Quote:
Original post by Dookie
Will it work fine with Windows XP, or is it more designed with Vista in mind?

Yes, both. It plays better with Vista's security system compared to 2005 (which had some problems there) and runs fine on XP as well (which is what I usually run it on, for no other reason than I've not felt the need to upgrade my desktop.)

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.