In general, I have always been interested in computer graphics. There is lots of different problems to solve, and if you like math/geometry then there really aren't many better ways to exercise your brain than working in this area. Once you know how to take geometry and project it onto an image, it can also be quite satisfying to do the opposite - start to investigate computer vision and understand how to take an image, and convert it back into a set of objects. A while back, I integrated Kinect support into Hieroglyph 3 for just such a reason - you can do some really interesting stuff with a color and a depth map of a scene, and even cooler stuff if you take that image sequence over time.

A natural extension of generating images for a monitor is to generate them for something like the Oculus Rift. This is actually not much different than regular computer graphics, except that you get the position and orientation of your camera from the Head Mounted Display (HMD), and then render the scene twice for two eyes. I also explored this by adding Oculus Rift support into Hieroglyph 3. It is really cool to play around with the technology and see how it works. Everyone talks about 'presence' with VR, and it really is a bit spooky how much you can get tricked into feeling like you are there instead of here.

However, in the end VR stuff basically takes you from a 2D monitor window into a virtual world, and wraps that virtual world all around you. The problem is, there is no in between - you can't see anything of the real world when you have the HMD on. This presents all kinds of problems, including the question about how do you interact with a virtual world if you can't see your own hands! There are lots of extremely smart people working on this very problem, many of them at Oculus I'm sure. But there is a solution to this problem already coming down the road, that might just change the very nature of how we approach interfacing to the devices all around us: The Hololens.



If you think about it, the Hololens combines both computer graphics and computer vision. You get to put computer generated objects into the real world using computer vision. Of course, we don't know yet how much access we will get to the underlying technology (although that answer is apparently at least partially coming at Build according to a few interviews) but it is really cool to think you can interact with the basic structure of the world around you - at the same time you are wearing the HMD.

That totally solves the challenge of how to interact with the world around you with the HMD on. I don't know how well it is implemented, or how low the latency is on the headset, but if it works as everyone seems to be reporting, then I can't wait to get my hands on one of the development kits for the Hololens. Consider the example Minecraft demo that Microsoft showed off:



Think of all the cool things you could do with access to the room's basic structure, and selecting overwriting of its contents. Have you ever played Portal? The opportunities are limitless...

What do you guys think - what will you do with this technology!?!?!