I'm in your same boat: trying to provide my own localizable string interface that's platform-independent.
I believe UTF-8 encoding is an encoding format for storing text that would multiple bytes to identify, yet it's meant to be backwards-compatible with ASCII. With that said, that front bit in each byte isn't used in calculating it's UNICODE value. Instead, it's used as a flag to determine if the next byte of data's back 7 bits are used to describe the localized character's unique value. These values, once calculated into normalized 32-bit (unsigned?) UNICODE encoding. Then, you'd have a bitmapped font that would represent each glyph in the font on the image by a 32-bit (again, unsigned?) value equal to the what would match up to your localized text's normalized UNICODE values.
I think that's in line with what Hodgman was saying above. With that being said, you'd want to store your text that you'd display onscreen in UTF-8 encoding, and use a library to read those UTF-8 strings, such as utf8proc, to convert it into a string of normalized UNICODE characters that'd you'd use as look-up values in your localized bitmapped font when rendering text to the screen.
Since we're storing text in UTF-8 and XML parsers typically expect its text to be stored as UTF-8 text, I store my strings in an XML schema like so:
Hello! Hola! New Game Nuevo Juego
I'd like to point out that my XML code doesn't appear here^ Looks like it was edited out :/
Then, in my code, I'd have a LocalizedPackage that'd load up an XML file of localized text, typically for an entire menu or for cut-scene dialog that would contain a collection of localized strings described by my LocalizedSting class. LocalizedPackage reads the XML file, and for each element in the XML, it create a LocalizedString instance. LocalizedString would then create an instance of LocalizedText that'd it hold for each element found. It'd read each element, use the 2-character code to determine which language it falls under, and label that LocalizedText with that language. the XML parser would read in the text, and tell it as UTF-8 string and convert it to normalized, an array/vector/list of unsigned long's.
Then, you'd do something like this in your code:
LocalizedPackage contains would keep track of the game's current language with this variable:
// in .h
static int currentLanguage;
// in .cpp
int LocalizedPackage::currentLanguage = LANGUAGE_ENGLISH; // set default language to English
LocalizedPackage would return the correct language's normalized UNICODE string that my FontString class would know how to interpret. Of course, if you wanted to provide localized text in a specific language regardless of the engine's current language, you could always do this:
You would want to provide lots of error-checking so that GetString() returns NULL if something's invalid, and have SetText() check if it's receiving non-NULL data...
I don't have this completely implemented yet, but I hope this gives you ideas!
EDIT: LocalizedPackage could be expanded to also load more than just localized text --images, sounds, music, etc.
Edited by Vincent_M, 16 May 2013 - 01:35 PM.