Jump to content

  • Log In with Google      Sign In   
  • Create Account

Squared'D's Journal

How about UNICODE and UTF-8

Posted by , 01 August 2013 - - - - - - · 1,936 views
utf8, unicode
My Thought Process

I've been thinking this over for a while. Right now, my game engine and GUI do most things in USC-2. USC-2 is a 16 bit version of UNICODE and it can encode the values for all of the most common UNICODE characters. I think these days, Windows supports UTF-16 as well. I started to use it because I do most of my work on Windows, and this fits perfectly with the Windows 16 bit wchar_t type. I had avoided UTF-8 because it's a variably-sized encoding which means some character will use 8 bits while others will use 16, 24, and at most 32. Because of this, I would no longer be able to easily use string[index] to get a specific character.

But lately, I've been wondering if I made a good decision. Post like this (link to comment) got me thinking. I've been so single platform focused that I haven't really considered some real problems. For one thing, wchar_t is 16 bits on Windows, but 32 on some other platforms. Also when using wchar_t and even with UTF-16 and UTF-32, you have to worry about endianess. So I've been considering using UTF-8 for all of my strings. This is possible because I use my own GUI anyway and will only have to convert with using API specific functions. If I can limit this, I shouldn't have any problems.

What is UTF8 and how is it encoded?
Now a lot of the information in this section comes from http://www.fileformat.info/info/unicode/utf8.htm so you should look at it for more detailed information, but I'll try to explain it the best I can. Just one thing, when I say character, I'm referring to the value of the character in unicode which goes from 1 until 1,112,064 (http://en.m.wikipedia.org/wiki/UTF-8) with zero which can be used as a string terminator.

Like I said before, UTF8 can encode characters into 1, 2, 3, or 4 bytes. 1 byte encodings are only for characters from 0 to 127 meaning if it's a 1 byte encoding it'll be equivilent to ASCII. 2 byte encodings are for characters from 128 to 2047. 3 byte encodings are for characters from 2048 to 65535 and 4 byte encodings are for characters from 65536 to 1,112,064. To understand how the encoding works, we'll need to examine the binary representation of each character's numeric value. To do this easily, I'll also use hexadecimal notation as 1 hexadecimal digit always corresponds to a 4 bit nibble. Here's a quick table.

Posted Image

So 2C (hexadecimal) = (2 X 16) + (12 X 1) = 48(decimal) and 0010 1100(binary)
I understand that many understand this already, but I want to make sure new programmers will be able to understand.

The UTF-8 Format
In UTF-8, the high-order bits in binary are important. UTF-8 works by using the leading high-order bits of the first byte to tell how many bytes were used to encode the value. For 8 bit encoding from 0 to 127, the high-order bit will always be zero. Because of this, if the high-order bit is zero, the byte will always be treated as a single byte encoding. Therefore, all single byte encodings have the following form: 0XXX XXXX
7 bits are available to code the number. Here is the format for all of the encodings:

Posted Image

Once you know the format of UTF-8, converting back and forth between it is fairly simple. To convert to UTF-8, you can easily see if it will encode to 1, 2, 3, or 4 bytes by checking the numerical range. Then copy the bits to the correct location. After a few weeks, I'll post a class that puts all of this togther and one that can convert between the different UNICODE formats.

Example Conversion
Let's try an example. I'm going to use hexadecimal value 1FACBD for this example. Now, I don't believe this is a real UNICODE character, but it'll help us see how to encode values. The number is greater than FFFF so it will require a 4-byte encoding. Let's see how it'll work. First, here's the value in binary.

Posted Image

This will be a 4-byte encoding so we'll need to use the following format.

Posted Image

Now converting to UTF-8 is as simple as copying the bits from right to left into the correct positions.

Posted Image

Like I said, UTF-8 is a fairly straight-forward format.

Questions and Final Thoughts
Now it's not difficult converting back and forth between UTF8, but is this really the best solution? Would it be better to continue to store things in a know string type and just be able to convert back and forth, using UTF8 only to store things in files? Maybe I could continue to use wide character strings in the native OS format and just and write some functions to do the conversions. If you have any preferred strategies, I'd like to hear them.

Project B - 2013-07-30 Update -- She Moves!

Posted by , in Project B 30 July 2013 - - - - - - · 679 views
animation, project b, indie
I just wanted to post a quick update. The female running animation is in. It still needs some tweaking, but it's starting to come along quite nicely now.

Posted Image

Project B - 2013-07-29 Update Animations

Posted by , 29 July 2013 - - - - - - · 663 views
project B, game development
Things have been going a little slower than I had originally expected, but things have really been picking up. I've got models loading and animating. Next I'll start putting levels back in, work on some networking code, and the add model attachments. Here's my updated schedule.

Posted Image

I'm also going to include the latest version of the female model with improved textures. The artist in my team have been working hard getting animations done. It's been a bit slow as they're getting use to working with each other though. Hopefully soon they'll have new walk and run animations for me to show.

Posted Image

The video below is actually an older model. It was used for Genesis SEED. I'm using it here as test data while the other models are being worked on. This model was created, textured, and animated by Sung Shin.

Project B - 2013-07-23 Update -- Models (with screen shots)

Posted by , 22 July 2013 - - - - - - · 638 views
project b, game, game development and 1 more...
Project B - 2013-07-23 Update -- Models (with screen shots)

Re-adding models has been more difficult than expected. I guess I've changed things more than I had originally thought. As I was working on the model system, I decided to knock a few things off of my to-do list. I use an component-based entity system. Before, the graphics data and the other information need to define the components for an entity were separate files. The component information was stored in an XML file. I've changed things so now everything will be kept in one binary file. The new binary format is flexible and can keep information for many kinds of components--even components that I haven't though of yet.

I've got models loading and saving, but I'm still working on animations. Here are some screen shots. The model was provided by our team artist.

This shot shows a basic model. There's normal mapping and some glow.
Posted Image

This shows the front and back of the model. The face is shinier than the body.

Posted Image

Posted Image

Here's my updated schedule. I'm behind and need to catch up in a hurry.

Posted Image

Project B - 2013-07-17 Update

Posted by , in Project B 16 July 2013 - - - - - - · 753 views
Project B, game, game programming and 2 more...
Project B - 2013-07-17 Update

Progress has been going well though a bit slower than I had expected. I'm still behind after spending more than a week working on this problem. (direct-x-11-problems-on-windows-8-laptop). I've gotten a lot of work done. You can take a look at the current state of things.

Posted Image

I've been building a model tool that will be used in my content pipeline. After I get real models into it, I'll post a video blog. Right now it allows me to configure shaders and textures. Here are a couple of screenshots. For me a skin is defined as a texture/shader combination to be used with a model.

Posted Image

Posted Image

Thanks for reading.

Doing It On Paper

Posted by , 14 July 2013 - - - - - - · 583 views

Doing It On Paper

Work has been going well these days, but I've decided to take a break from the standard progress report and tell one of my lessons learned stories.

In school programming courses (high school or college), students are often forced to look at programs, trace, the flow of execution and write the programs output. This seems like a trivial unimportant exercise while you're doing it, but the skill can come in handy. This happened to me last year when I was coding Genesis SEED, my teams 3D multi-player IGF entry last year. The game has 5 basic modes: single player, multi-player server, multi-player client(with client-side prediction), peer-to-peer host, and basic peer. It was a year ago and I don't remember what the actual problem was, but basically the game worked in some modes but not in all. I figured that I made a mistake and somewhere something was missing. I must have forgotten a step somewhere. That's why I made the following:

Posted Image

I found that rummaging through papers while I was looking for something else. I thought it'd be cool to show to everyone. By tracing out the code I was able to find out what was missing. I didn't even have to do it for every mode. This is just one example of paper to the rescue. Sometimes when we debug, we want to quickly get into things and start changing code, but often what we really need to do is sit down and write it out on paper and examine it.

Project B - 2013-07-08 Update

Posted by , in Project B 08 July 2013 - - - - - - · 611 views
Project B, game, game development
2013-07-08 Update

Progress has been going at a good pace, albeit a little more slowly than I had thought. The code that the entire engine runs on has changed, and I've been trying to slowly reintegrate everything. A lot of the low-level graphics code is done. That last little tech demo that I did may not have looked like much, but there were a lot of things going on under the hood. Shaders, textures, post-processing, rendering to a texture, rendering states, blend states, and basic start up code are all in. I've now been developing some of the more high level engine systems. These are the systems that game itself will touch.

I already know what things too make as I'm making exactly what I did before, but it's not as easy as it sounds. For one, as I've been writing this code, I've found that I've also needed to write the code that connects everything together. For example, the last demo that I made uses a temporary framework, but I've had to upgrade that a bit so I it behaves more like than final stuff.

I've also be taking some time to try to upgrade some of the engine's designs. From my own personal lessons learned and also from reading articles, blogs, and forum post, I've decided to change some things especially when it comes to asset management. I also want my future code to be multi-platform, so I have to make sure Direct X specific stuff doesn't rear it's ugly head in my higher level things.

One other thing that has cause a little bit of a slow down is I don't just copy and paste code from the original engine. I examine it. Because of this, I've found some bugs. I had a bug in my old game whereby, objects would look dark if you looked at them from certain angles. It turns out that the code that sets some of the shader constants wasn't getting the correct camera position. It was always getting (0.0, 0.0, 0.0).

Posted Image

I'm a little behind schedule, so I've updated things a bit to show where I am.

Project B - More On The Update and Post-Processing Framework

Posted by , 03 July 2013 - - - - - - · 1,956 views
game, post-processing, glow
Engine Post-Processing Framework

I feel really bad about my last post. I feel like I made it too quickly and I did a poor job of explaining things. Because of this, I'm going to try to explain things a little better.I want to give more details of what is in the demo that I posted earlier. Though I talk a lot about the GUI, actually the first thing that I worked on was the post-processing framework. What is post-processing, basically with it, you can achieve all sorts of effects by running the entire completed scene through another set of shaders. You can use it to achieve effects like glow, motion blur, and many other things.

Here's an example of glow being created as a post-processing effect. From Genesis SEED, out IGF entry last year

Posted Image

This was the first system that I worked on because I wanted to fix a bug in it. I also wanted to design a really robust framework capable of handling many different kinds of effects.

How to do Post-Processing

The basic idea of post-processing is first render the screen to an auxiliary buffer instead of the back buffer. This auxiliary buffer can then be set as a texture to a shader. Then render a full screen quad of the texture using some effect shader. To perform different effects, you may need more than one auxiliary buffer. This is shown below.

Posted Image

The demo that I included in my last post used a glow effect. To perform glow, it uses the alpha channel to determine which areas should glow. The scene is first rendered to an auxiliary buffer. Then I use a 1/4 size buffer and render the entire scene to that buffer using a filter to only write pixels whose alpha value is set. Then I perform a vertical blur and then a horizontal blur. The final blurred texture is combined with the original image and then rendered to the back buffer. If you download the demo from my last post, you can take a look at the HLSL shaders.

Making a good post-processing framework isn't that hard. There's a good article on the subject here.

Some Cons

You have to be careful though. As you will be rendering the entire screen multiple times, you can lose a lot of speed. One way to speed things up is by using 1/2, 1/4 or even 1/8 size intermediary auxiliary buffers.

I hope this was useful. Check out my previous post to download the demo and see this and my GUI in action.

Previous Post:
Here are some screen shots.

Posted Image

Posted Image

Posted Image

Project B - 2013-07-03 Update (with download)

Posted by , in Project B 03 July 2013 - - - - - - · 922 views
project b, game, game programming
Project B Update

Progress has been going well. I'm kinda sorta keeping up with my schedule. I got a little behind because of a bug that shows up on some laptops (WIN 8 LAPTOP BUG) so I had to postpone some GUI things. I originally used the DXUT GUI for all of my GUI needs, but I've since re-made the GUI from scratch in my effort to one day go multi-platform. I won't be able to add loading from files until the end of the month.

Here's my updated schedule.

Posted Image

From tomorrow, I'll start working on shaders and textures. Yes, the program is using shaders and textures now, but I need to re-build the higher-level system that my previous game used. My hope is that once this is finished, the systems will for the most part resemble the old engine's interfaces. That way I'll be able to change the game to the new framework without much fuss. This is a big job and I'll need to do a lot of testing a long the way.

I want to submit the final project to IGF again so testing is important. If you want to help me test by downloading the current framework test, I'd greatly appreciate it. (Sorry no source code)

DOWNLOAD NOTES: If you download the test, it'll first prompt you if you want to use DirectX 9. If you don't have DX11 hardware, or Windows 7 or 8, you must click "Yes". I think the next prompt will ask for DX11. If you say no, it will still use DX11, but only the DX10 level shaders.

I may be able to post again this week, but it'll probably be next week.after I have something to talk about.

Thanks for reading.

Attached Files

Project B - New Project

Posted by , in Project B 30 June 2013 - - - - - - · 848 views
game, game project
Project B

Hello all. I've been working hard on putting the graphics-less game together and I'm going to continue to work on it, but work will go a little slower as I'm starting to work on a new game project. I've already developed a 3D multi-player game engine that was used for my last IGF entry shown here (Genesis SEED), but these days I've been doing some major improvements and updating the rendering system. The project was in limbo so I started doing other things. Now with this new project, I need to quickly get my improvements up and running. Basically, the old system was DX 9 only, but I've been building a new system that I hope will give me multi-platform capabilities in the future. Currently it supports DX 9 and DX 11. Here's a list of the things that I need to finish by August. While I the old project was in limbo, I also started building a new GUI system. I want to finish it and use it with this new game. I'll try to provide regular updates here, maybe even some videos that detail my progress.

Here's my task list:

Posted Image

I picked up the color-coded system back when I was working for the USDA. I use to do contract management and IT security while I was there. The project managers for the database systems would use a similar color-coded system. I like it because it easily lets me see what I need to do. And I hate seeing red. That motivates me to work faster. Right now, there is too much red.

I will update this blog weekly. I'd like to submit this game to IGF so I'll probably post some test framework code for the public to try out.

Recent Comments

January 2017 »

2223 24 25262728