Diarmid Campbell

Camera-Based Gaming: How to Do It and Why you Would Want to

Making a game that uses a camera is difficult – not because of the game itself, but because of the treatment you must apply in order to get something interesting from the camera. If you think that understanding an image is easy, think twice. Your brain does many things under the hood, and before you can do anything yourself, you must understand how everything works.

Here is a typical problem with the way our brain works: we automatically identify colors with respect to their environment. The context is important to us, but a computer will have difficulties understanding it. This leads to many problems: for example, color recognition is notoriously difficult to perform.

To identify a player in the capture area, many techniques exist: for example, image subtraction (take a reference image without the player, and bring the player back into the camera: everything different is supposed to be the player; unfortunately, this technique is very fragile). Another algorithm: sequential frame comparison. A motion buffer computes the difference between two frames. Unfortunately, this technique captures every movement, not just the player's movements. UI motion buttons can use this technique. To use it effectively, a good idea is to let the software accumulate motion before the action is triggered. Vector buttons also use this algorithm. The motion is understood by the program and transformed into a motion vector.

Optical flow is based upon interest points that will be easily found in the next image. By computing the difference between their positions in subsequent frames, we can find motion vectors as well. Unfortunately, it's very difficult to differentiate between different movements – a rotational move is a rotation move, no matter what part of the body performs it.

Creature Feature (available on the PSN to download) is a game that implements many of these UI features. The goal of the game is quite similar to the goal of lemmings – except that the movements of your body attract creatures you must save. It uses optical flows to implement scroll bars and motion buttons to implement the traditional UI buttons.

Good input devices feature a unique set of features:

• They are flexible and rich; a camera is very flexible – the number of parameters to take into account is the number of pixels of the device (its resolution).
• They are non-abstract: an action on the controller is directly mapped to an action in the game (swing the wii controller and you'll hit this tennis ball).
• They are robust and responsive: false input shall be avoided, but it's best to respond to input immediately.

That led us to the best show I ever seen in a conference. Imagine a talker taking two pompons (a red one and a violet one; these colors can easily be tracked by a camera by using hue correlation) to play a dance game (Pompon Party, to be released on PS2) where the pompons are tracked by an EyeToy camera. And he must sing too. That was priceless. But let’s go back to work.

Diarmid gave us many examples of EyeToy-based input devices (polystyrene ball, a simple drawing (call this "sketch technology"), and so on).

Of course, you don't always need a specific help to use the EyeToy. You can try to track the head (which is good; as Diarmid said, "everybody has a head, so we don't need to ship it with the game"). Unfortunately, head tracking is difficult. Sony Japan is currently trying to improve the state of art – but robustness is still to be enhanced.

Using cameras allows you to make augmented reality games. Step one: capture the frame, threshold it, find edges, test for the presence of a marker, calculate orientation, and replace the marker with a 3D object.

According to our host, the EyeToy brought physical gaming to public attention (and the Wii made it mainstream). Camera-based games can bring many things into the game arena because of their ease of use. They also help immersion of the player in the game. And they have a bright future: researchers are already working on the next generation of camera devices: 3D cameras. Some photos courtesy of Connection Events.