I'm trying to get a handle on where the games industry stands on AI speech in terms of allowing NPC's to act more human. In most games now, voice actors are used to keep the story and experience with NPC's somewhat linear. Are any companies looking at expanding their approach to AI? What about using something like Watson in a game environment? Is it possible right now - why or why not?

Text to speech (TTS) isn't an AI problem.

I'm thinking of moving this in the site, but I'm not sure where it fits. It isn't an AI issue, isn't really a programming issue at all. It might fit better in design, but it isn't so much a design issue either. Oh well. :rolleyes:

The big problem is that TTS engines do not have a good range of emotion. Some systems allow the teams to tag them with some vocal effects. You can encode volume going up or down, pitch up and down, and a few rough elements like that.

But how do you leverage a TTS engine -- even one like Siri since that is popular -- make an NPC voice call out like they're in a market as a seller? Or like they're injured? Or like they deeply care that their child is lost? Or like their family has been killed?

Basically, how can you make a voice like Siri sound like a demon has killed their family? The simple answer is that you hire a voice actor.

Text to speech (TTS) isn't an AI problem.

Since he mentioned the "experience" and "story" of NPCs, I think he's talking more than just audio.

The big problem is that TTS engines do not have a good range of emotion. Some systems allow the teams to tag them with some vocal effects. You can encode volume going up or down, pitch up and down, and a few rough elements like that.
But how do you leverage a TTS engine -- even one like Siri since that is popular -- make an NPC voice call out like they're in a market as a seller? Or like they're injured? Or like they deeply care that their child is lost? Or like their family has been killed?

Basically, how can you make a voice like Siri sound like a demon has killed their family? The simple answer is that you hire a voice actor.

I think it would be possible to give TTS engines more emotions and variation and "life". I remember seeing six or seven years ago a program for taking Microphone speech and transforming your voice before sending it to whatever game you were playing, to change your voice in real-time for in-game VOIP communication. Things like making your voice sound like a robot or demon or (ostensibly) the member of the opposite gender - really basic audio filter stuff. I'm sure if there was enough market motivation (and game NPCs are a good example), it could be rapidly improved. It might already be improved significantly, since I saw that a half-decade ago.

(Personally, I was thinking that might be useful in a microphone-required game, for hiding or reducing the squeakiness of prepubescent voices, to reduce immersion-breaking, as well as prevent the harrassment that I've observed when a kid tries to play a real team-focused FPS. Would possibly also be useful to remove noise and repair sound from cheap microphones)

You'd likely have to add annotations to your NPC text to indicate what emotions should be used, but that's not much of a problem.

I agree fully about the constraint that voice acting has placed on content generation, and look forward to what people come up with when that bottleneck is removed.

Neural architectures are really advancing TTS by leaps and bounds. But I give it about 2-3 years before it's ready for game use.

A lot of voice acting isn't emotionally quite right for a game's scene anyway, in part because the voice actor doesn't always know all the context, might not be recording at the same time as the other actors, is expecting other content that gets left on the cutting room floor... In a few years we might be preferring synthesized voice, because you can tweak it for content and emotion as the game changes without re-hiring the voice actor.

If you want to play around with making a TTS-based game using a state-of-the-art speech system, I recommend playing around with Amazon's Alexa Skills Kit. It's pretty easy, you can use AWS Lambda as your backend rather than set up your own server, and you could probably make a simple game -- speech in, speech out -- in a few days. Intonation isn't Alexa's strong suit, though -- especially compared to Wavenet, which completely raised the bar -- so a game like Hodgman suggests where she's actually playing as some sort of AI will be more realistic.

(It's also interesting to note that Amazon has been rolling out a lot of gamedev products like Lumberyard and GameLift. It's probably only a matter of time before Alexa speech tech finds its way into Lumberyard.)

The game 5089 does this kind of. You submit quotes you want robots to say on the steam forum, the dev runs it through TTS software, and puts in the game.

So I find this really really fascinating. I also have a dream of a game that involves Proc-gen speech. The WaveNet is absolutely brilliant, and I can scarecely understand the underlying techniques, but I'd love to see more of that. I'd love to have access to that as a tool. I'm not sure how that works with stuff that's still being actively researched though. Still, crazy promising. You could possibly train different emotions into the system and have it sample those emotions instead when creating voices, or sample multiple at once to make things really interesting.

I'll look into Alexa. I was honestly considering making a proc-gen dialogue game that was about robots, so that normal TTS systems could be used without breaking immersion into tiny pieces. It'd be crazy, a world that was like a giant menu in some ways. That could be fun, a little robot society.

At the end of the day. There is still something to actual voice actors that can't really be duplicated. I do wonder though what would happen if you gave a voice actor a WaveNet tool to craft their own procedural performer? That would be really epic.

And its potential for misuse is scary.

And this too. A convincing human digitized voice, or being able to take someone's voice and easily make it say something else, that's a pretty significant change to life as we know it.