• entries
    121
  • comments
    278
  • views
    156085

Revolution, Part 1: Audio

Sign in to follow this  

452 views

A friend of mine has been working very hard on a project which involves two revolutionary technical innovations. I am not usually the kind to care for revolutions, but I honestly believe that both of these are really, really significant. I'm actually hoping to join up and help make these things into a proper product, in fact. These are things you have not seen in games, and he has it working in prototype form on a mobile platform. That's unusual in this industry.

Here,
">listen to this Youtube video while reading. You need headphones for this, which is why the tech is launching on the iPhone first. The video is not ours, but it is a very good demonstration. Just listen and read. Remember, it doesn't work without headphones -- if you don't have any, it might actually be worth reading this only after you've found some.

The underlying principal is binaural recording. (This is not the revolutionary bit, and has been around for quite a long time.) Games have had 3D audio for ages, so that is in itself nothing to get excited about. Current 3D audio basically works by modifying channel volumes for playback of a mono sound in order to simulate a 3D space. It works alright if you have a 5.1 setup, but it's not terribly effective in stereo and in general the effect is a bit weak. Binaural recording, however, is a method of recording sounds with a pair of microphones and an actual head model that attempts to produce a stereo sound that simulates what our ears hear. You need headphones because of the recording methodology, and if you're listening to the video I linked, you're probably spazzing out right now.

There is a catch to all this, which is that nobody can synthesize it. The sound is recorded by physically placing it relative to the head, so you can't go back later and place it at an arbitrary location. (Some people have pointed out that there are processors and algorithms that try, but they are expensive and don't really work well.) That's essentially why it's never showed up in games --although headphones-only isn't a thrilling restriction, either. Still enjoying the barber?

Here is his binaural recording. (It's 8 seconds, just pause the barber.) There's one key difference, though. That's not a binaural recording of a sound being moved in front of a recording head. It is done in real-time. (This is the revolutionary bit.) This friend of mine has figured out how to do it. The original implementation worked very well but required a lot of memory and processing power. But the current system is efficient enough to fit on the iPhone. I've seen and heard the demo, working in real time off an iPod Touch. It works well enough to make your skin crawl, like those scissors are probably doing right now if you're still listening to the barber.

I can't really say too much about how he's pulled it off, because we think it's kind of a big deal. You'll see iPhone game releases with the technology later this year, and hopefully by early to mid next year we'll be licensing an actual SDK for whatever platform you might care to use. We're fairly confident this is technology people will want, and hey, it wouldn't hurt to forward this post around the office.
Sign in to follow this  


9 Comments


Recommended Comments

Ok, I am officially creeped out. o_O

The ghost doesn't sound quite as creepily real, but I'm guessing that's due to it being only 8 seconds. I can imagine there being a whole host of unique iPhone games that could come out of this.

Share this comment


Link to comment
That is amazing! Please, can you create a clip longer than the ghost?

I mean, the most obvious application is horror games. This kind of synthesized sound can easily double the horror quality of a game, if not more.

Big thumbs up to your friend.

Share this comment


Link to comment
It sounds like a good demo, though I'm not convinced by its revolutionary claims; binaural recordings work thanks to the head-related transfer function, and there are a number of existing hardware and software HRTF implementations (eg Dolby Headphone or ffdshow's HRTF mixer setting). The results are not as good when coming from mixed sound rather than the original positional audio, though.

Edit: That sounds needlessly negative; don't get me wrong - I love this technology and would love to see it used more. [grin]

Share this comment


Link to comment
Quote:
Original post by benryves
It sounds like a good demo, though I'm not convinced by its revolutionary claims; binaural recordings work thanks to the head-related transfer function, and there are a number of existing hardware and software HRTF implementations (eg Dolby Headphone or ffdshow's HRTF mixer setting). The results are not as good when coming from mixed sound rather than the original positional audio, though.

Edit: That sounds needlessly negative; don't get me wrong - I love this technology and would love to see it used more. [grin]
Yeah, a quick bit of googling even found a paper on a real-time DSP implementation from 1997.

Still, incredibly cool work getting it on the iPhone! Whether the tech is revolutionary or not, what will be revolutionary is having it on a portable gaming device.

Share this comment


Link to comment
It's not my engineering work, so I am not really equipped to go into details, but it's my understanding that this provides better and cheaper results than HRTF based work. The friend studied them in detail; I know next to nothing about them. My main question is, if they have this stuff that can do it, where is it? I would think there'd be at least the occasional game using it. (And reading the wiki page, they sound hideously complex and limited...)

Plus, this is cheap and requires no special hardware support -- I'm considering porting it to the DS if the homebrew stuff allows for it. And of course it's a very strong offering in the mobile space, where most players are likely to have headphones.

Share this comment


Link to comment
Quote:
Original post by Promit
My main question is, if they have this stuff that can do it, where is it? I would think there'd be at least the occasional game using it.
The most popular use of this technology currently seems to be in downmixing surround sound to stereo headphones. Audio APIs (eg FMOD and DirectSound) offer HRTF to varying levels of quality, though it doesn't seem to be fashionable to enable it by default.
Quote:
Plus, this is cheap and requires no special hardware support -- I'm considering porting it to the DS if the homebrew stuff allows for it. And of course it's a very strong offering in the mobile space, where most players are likely to have headphones.
This may be why it's not so popular on desktop PCs or consoles, where people will be generally playing on speakers.

Share this comment


Link to comment
Quote:
Original post by Jason Z
So what is the other revolutionary technology???
I don't have a good demo of it in hand yet. Hence "Part 1"...hopefully soon I'll have the other half.

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now