I'm interested in item 4. We would have to build a parametric model of the speaker so we can then tweak the parameters to generate many new speakers.

One way to make it happen would be to train something like WaveNet on multiple speakers, conditioning on a small vector that encodes information about the speaker. Part of that vector could describe gender and age, part of the vector could be a learned embedding of the accent (so we can label the speaker as having "British English accent", or "Southern American accent", etc.) and the rest of the vector would be a learned embedding of the speaker. We could then generate new speakers by specifying gender, age and accent, and then generating some random numbers for the specific speaker.

While we are at it, the same conditioning mechanism could be used to indicate some rough emotion of the utterance, addressing item 3 to some extent.

Does this sound feasible?

2 hours ago, Aharon Satt said:

What would allow game developers to start benefiting from the TTS technology?

* A system that translates to other languages

* A virtual philosopher that has something to say

But realistically - personally i have high interest. Looking up for something like a library once a year but never found any. (I guess the industry has only little interest: If we need a writer to write any text, we can hire a voice actor as well)

Some ideas:

Some language that discribes intonations, feelings and stuff, e.g. "<frightened 5> Help! The monster! <whispering 2> Hide under the desc!"

Eventually a musician could also add this data in real time by playing along with a keyboard, controlling both pitch, mood etc. (I remember guitarist Steve Vai imitating spoken speech on guitar, and it really works well - he can express those things very accurate, although you can not understand the words themselves.)

We need laaughter and screams, not just speech.

Generating data usable for automatic lipsync and facial animation would be nice bonus.

We might be happy with something that acts as a placeholder during production, and will be replaced by voice actors when everything is done. For now at least...

Some games have used TTS for quite some time.  

It isn't fancy, but consider how Animal Crossing has done that for more than a decade. The text doesn't sound like professional voice acting, but when you've played for a while you can understand every utterance.

There are many games where TTS is thematically appropriate. Games set in computerized worlds would be a great natural fit.   Also less natural, but many text-based games have used it for years, particularly with vision-impaired players.

So I think there is an item 0 on your list, the context of the synthesized speech.  

On 1/18/2018 at 9:12 AM, Aharon Satt said:

I am curious to find out what is your opinion of this.

My personal opinion is complete indifference about TTS.  As I wrote above it depends entirely on the game.

Some games with enormous budgets and high-end designs require voice acting that truly does need professional voice actors, speech coaches, and high-end recording studios.

Other games can use -- and already use -- rudimentary audio tables for TTS that fits their game design wonderfully.

Naturally the course of software is that it offers more options, higher audio fidelity, and more choices for encoding and use, but that is not an issue of the tech being suitable for gaming in general.

To me the actual issue is the cost of a tech solution versus the cost and the (potentially enormous) benefits of the customized voice acting.  If you've got the budget to create a game worth tens of thousands of lines of voice acting, or even hundreds of thousands of lines of voice acting, then you've got a budget to use a wide range of voice synthesis systems. At that point you can study what is out there, the costs involved, the benefits of the actual actors, and decide based on the details of the game itself.

I already gave the example of Animal Crossing that has been using very simple TTS for nearly two decades with great appeal.

For another example, when I worked on Tiger Woods golf there was an enormous corpus of audio from the announcers, carefully edited so it could be pieced together based on any situation. There could be comments about the distances, the lie of the ball, the quality of the shot, the wind and other environmental factors, and much more. The benefit of having specific well-known individuals far outweighed the costs. Even if we could have used a high end TTS system that made realistic tones and inflections it would have been a terrible mix.

On The Sims series we had a wide range of audio.  We had Simlish clips put together by real humans speaking nonsense syllables, but these could have been strung together by anybody and may have been a good fit for some TTS systems today. However, since the middle of the original Sims 1 series they have brought in big-name musicians to record their own music in Simlish.  Bringing in Katy Perry, Lily Allen, Depeche Mode, Annaca, and other singers is not a choice about the quality of audio technology, but about the singers themselves. Bringing in the big-name singer is far more valuable to the game than any TTS system would be.

In the Littlest PetShop series series we used a system similar to what Animal Crossing used. Players commented on how they loved it, it gave a great style to the game that a Simlish-style speech engine could not have used.

These depend entirely on the game, not the quality of the TTS system.

Some games have more than enough with yesterday's technology. Some games can use today's technology.  And some games would not accept the TTS technology no matter how advanced it became, because the voice actors themselves are necessary.  It is all about the games, not the tech.