I'm just wondering what challenges I'll face. It seems easy enough to me but maybe I'm forgetting something. Is it just a case of encoding the captured data, sending that over network as fast as possible and having clients decode it and play?

It's not quite that simple

You need to deal with a couple main issues:

You'll want to send your data UDP (not TCP). TCP is fine for sending files, etc but is not well suited for real-time communications.

1) when you send data over the internet via UDP, you're not guaranteed that the client receiving data will get in the same order that you sent it. I.e. if you send 5 little packets of audio data, A,B,C,D,E, they might be received in the order B,C,E,A,D. For that reason Internet voice systems put little sequence numbers on their data packets. When receiving data, the receiving client saves up a few packets before playing them-- this gives the client a chance to put them into the right order. FYI, that is usually called a "jitter buffer."

2) You have to account for a packet never arriving at all. In the example above, packet "C" may never arrive--ever. So your software has to be clever about what to do in that situation. Typically the client will wait a certain amoutn of time waiting for that 'lost' packet, and then just decide it will never arrive. Then it will usually just play one of the packets twice (eg A B B D E) to make up for the lost packet.

You face these challenges even if you don't compress the audio data at all-- Dealing with these 2 issues are among the main ways that bad VOIP systems sound worse than good VOIP systems.

You'll have to manage the "robustness vs latency" issue. That is, the bigger your buffers, the less likely your sound will break up or crackle, but the longer your latency (delay from the time person A speaks and person B hears) will be. After network latency, you really want to shoot for end-to end latency of less than 200ms (including the latency introduced by the jitter buffers). Much more than that, and conversation gets difficult/annoying

Opus is a good choice. It has "wide band" (High fidelity) modes for speech.

Brian Schmidt

Brian Schmidt Studios

I think I'm just overthinking this.

I'd say your are correct :)...

While it is quite possible to create a sophisticated system that looks at network QoS (Quality of Service) and does run-time analysis to give each player the best quality, dependent on their own network with server-side encoding, in practice, it's way easier to just decide on what quality you want to deliver and always use that setting.

So encode on the sender's system at whatever you decide is the quality you want, send it up to the server and let the server broadcast it out to the people at the other end of the 'phone.' (or just go peer-to-peer and bypass the server if you don't need large #'s of people to hear)

I'm not familiar with the details/options of Opus. But in general, there are 2 things that will decide the audio fidellity: Sampling Rate of the sound itself and the compression ratio. The sampling rate should be at least 16kHz for "high quality" speech. (a regular telephone is 8kHz), and I'd say that anything over 24kHz for in-game chat is overkill.

Note that "sampling rate" will generally be about twice the 'bandwidth', so the -bandwidth option might be controlling the sampling rate. as for -loss, that probably directly or indirectly controls the compression ratio. The less compression, the more natural the voice will sound. at extreme compression ratios, you often get that warbly/watery/robotic sound to speech.

Hope that helps!

Brian

A high-quality encoder will pretty much always improve the experience compared to "raw" data. The reason is that bandwidth is *never* unlimited -- there's always going to be more conversations, more movie streams, more large file downloads, and more fridges calling the grocery to tell them you're out of milk.

The two things you need to solve are:

1) Topology -- peer to peer, or server-bounced?

2) Network and sound card differences.

To do peer-to-peer, you're going to need a NAT introducer in your server. There are links in the FAQ about this, but it's not entirely simple to set up. Going through a central server is much easier, and more robust for end users, but may add some latency, and will certainly add bandwidth usage to the server.

Secondly, different networks have different amounts of jitter, and different sound cards play out at slightly different sampling frequencies! Even if your sound card says it's doing 48 kHz sample rate, that may actually be 47,950 Hz, or 48,050, or, due to the mad drive to the lowest price, even as bad as 47,500 or 48,500 Hz. The reason this matters is that the sender may be producing samples at 48,500 samples per second (or some fraction thereof) and the sender will play out at 47,500 samples per second, and for each second continually played, the receiver will fall 1,000 samples behind. After 48 seconds of reception, the receiver will be a whole second behind in playback!

The fix for both of these is to keep a window of allowed play-out latency. When a packet with data comes in, put it in the queue, but don't necessarily immediately play it out. Decide on some lowest amount of data you need to be able to play out (say, 80 ms,) and some highest amount of data you will accept (say, 200 ms,) and then start playing when you get to the lowest amount. If each packet is 30 ms of data, you will start playing out when you have three packets. Then, as you play out, if you ever find that there's not enough data, you stop playing out until you have enough data (80 ms) again. Similarly, if you get additional data so you have more than 200 ms of data, drop all the older, unplayed data, until you're down at about (200+80)/2 == 140 milliseconds of data.

And, in a good sound implementation, you don't actually start or stop the playback stream; you keep the playback stream going, but "stopping playing" means you generate zeros for the play-out. That part is more related to sound management practices than networking, though :-)