Jump to content
  • Advertisement
Sign in to follow this  
  • entries
  • comments
  • views

Designing a Gesture Input System

Sign in to follow this  


A big focus in Masters will be the playeraEUR(TM)s ability to cast spells. In order to do so, the player must draw a gesture into the air, the game will recognize the gesture and cast the appropriate spell. For gesture recognition we make use of a custom-coded vanilla neural network. This is a machine learning algorithm that allows the system to learn any number of things, in our case the shape of spell gestures.

This works by first training the network with lots of example shapes. Once the network is trained, it can then recognize player input correctly. The great thing about this is that the training phase happens during development. So the game ships with a pre-trained network that allows for spot-on gesture recognition. Also, with a system like this in place, we may allow players to define their own spell shapes later in the game.

On the technical side, there are a few considerations that had to be worked out. Some of the more obvious pitfalls are that players wonaEUR(TM)t all draw their shapes in the same size or in the same position. This is easy to fix through normalization.

But in order to allow for perfectly smooth, frustration-free gameplay, we also want players to be able to draw their shapes in any direction, clockwise or counter-clockwise, and to start drawing closed shapes (such as a square) from any point (i.e. not force the player to start drawing a square from one of the corners). While some of these things were a bit tricky to implement, in the end we pulled it off without too many issues.


When we were done coding the algorithms for recognizing user gestures, we faced a big problem: gibberish input. Machine learning algorithms like the one we use are trained to recognize a given number of things. Our neural network assumes that whatever you draw is one of the spell shapes that the network was trained with.

To give an example: you may draw the number 8, and if this shape was not part of the training set of shapes, the network may end up being convinced that you just drew a circle. Because out of all the shapes the network learned, a circle may be the closest shape to the one you drew. In order to solve this, we had to implement a rejection mechanism that detects unknown shapes.

This is essentially a separate filter thataEUR(TM)s applied after the neural network outputs its prediction. We run a few tests figuring out whether the networkaEUR(TM)s guess makes sense given the user input, and if it does not we tell the player that their spell attempt failed. Getting back to the example from before, the network would tell us that the player drew a circle. We would run our filters and find out that the 8 that was drawn is too different from a circle (the closest known shape) and fail the spell.

We find machine learning a fascinating concept in general, and since our villains are supposed to be very powerful iconic characters, we may use machine learning algorithms for enemy AI, too. This would allow enemies to improve during the course of the game, adapting to the useraEUR(TM)s play style. No promises on this one though, sometimes aEURoefaking itaEUR? ends up being the better choice, time will tell. And by the way, a bit of aEURoefaking itaEUR? canaEUR(TM)t ever be avoided, because an AI that adapts perfectly to the player would have to be dumbed down in order to be beatable at all.


So much about the technical side of our gesture recognition system. Of course, these technical aspects and inner workings of the game are not relevant to players at all. They simply want to have fun, so we must provide them with a system thataEUR(TM)s both challenging and rewarding without being frustrating. While things are not set in stone yet, our current plan is to allow players to aEURoewriteaEUR? spells using a very simple aEURoesentenceaEUR? structure. The image shows you what this might look like.

Essentially, we want to reuse gestures frequently so that players can get accustomed to them and learn them in a fun way. Early on in the game, players may only be required to draw the gesture for aEURoeFireaEUR?, which is the middle square of Example I in the image. Doing so may generate a little flame in the playeraEUR(TM)s hand that might be used to light up a dark environment. Later, gestures can be extended and combined in many different ways. So the fire gesture might be extended into aEURoeFire-PushaEUR? to launch a fire ball or aEURoeFire-Push-BurstaEUR? to cause a big explosion.

We will have to do a lot of playtesting to find out how much complexity we can demand of players without things becoming frustrating. We donaEUR(TM)t want to overcomplicate the system, but we do want to give players a sense of achievement when they successfully cast a powerful spell.

This blog entry is cross posted from our main blog at http://masters-game.com/

Sign in to follow this  


Recommended Comments

That's interesting, I hadn't thought about needing an extra step to reject unknown shapes - thanks for sharing!

Share this comment

Link to comment

That's interesting, I hadn't thought about needing an extra step to reject unknown shapes - thanks for sharing!

Yes, that took us by surprise, too. We were so happy when we tested the network with various shapes, and it recognized them really well, even when drawn sloppily.

But when we started drawing shapes that it had not been trained with, we realized that the network cannot recognize "unknown shape" and instead spits out the closest known shape, sometimes with ridiculously high confidence values (98%).

Share this comment

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!