Hello everyone! I've been trying to find some resources on syncing audio and video in games and haven't come up with much. As background, I've written an emulator (Nostalgia, an Intellivision emulator) and I'm currently doing a complete re-write of the DirectX video section (it used to be DirectDraw). While I'm at it, I want to work on my synching of audio and video, since I've always felt it was just OK. Any suggestions on what I could be doing differently are very welcome. The emulator's engine is providing sound samples on-the-fly at 22500hz. It provides video frames at just a hair under 60hz (something like 59.5fps). So I'm keying off the sound to throttle. I let the whole system run as fast as it can. If everything is running such that it can produce sound samples faster than 22500hz, I keep pushing sound samples into the DS8 buffer until I reach a point that's N samples ahead of where I should be. I then wait for the play pointer to catch up to N/2 samples ahead of real-time, let the engine go and repeat. This works fine if the video system is fast enough to keep up. The problem is when my D3DDevice->Present() is stuck waiting for a VTrace. Then the audio gets behind and I have to move the play pointer back in time some so I don't end up playing stale buffers. This obviously results in choppy sound, and I'm still displaying video frames at less than the 60fps I should be. There must be some happy medium to get this right (perhaps using a separate throttling system?), but I haven't been able to come up with anything on my own. Any resources you fine folks can point me to will be greatly appreciated! Thanks, -Joe

The typical "real world" solution is to play audio in a separate thread. That way, you do all the actual loading and buffering at a totally separate pace from the actual game logic and graphics.

If you haven't done any multithreading it can be a little intimidating at first, but it's a good skill to master in any case, and definitely one of the more elegant ways to solve this kind of problem.

Quote:Original post by ApochPiQ
If you haven't done any multithreading it can be a little intimidating at first, but it's a good skill to master in any case, and definitely one of the more elegant ways to solve this kind of problem.

I certainly have no fear of multithreading - I've been writing threaded programs for over 6 years now - I'm just having some trouble wrapping my mind around the logic of synchrnozing 3 asynchronous processes:

My emulator engine, which produces the audio samples and video frames
DirectSound buffering, running at 22500hz
Direct3D flipping, running at a shade less than 60hz

So I was just hoping there were some examples of similar things or best practices that I could derive some ideas from.

Thanks,

-Joe

Not really sure why you want video at 60 FPS. Most typical video rates are signifcantly lower.

Anyway, one thread simply streams a video file from disk. You can usually decode both the audio and video from this one thread. The audio gets pushed into a ringbuffer for use in the playback thread. The main render thread simply checks the decoding thread for a ready picture frame for it to blit. Now, if the video starts to get behind the audio, you should drop some video frames (not render them) - it is much better than trying to drop/add audio frames (inserting blank audio or cutting audio clips out sounds really irritating, while missing a few pictures, or a paused video is not as bad).

At least, this is how I do it in my theora video player :D

Quote:Original post by pjcast
Not really sure why you want video at 60 FPS. Most typical video rates are signifcantly lower.

Because this is the rate that was provided by the hardware I'm emulating. When I drop frames to go to say, 30fps, the video looks wrong and people complain :) I'm getting about 42fps with everything running, and it does look a bit "off".

Quote:Original post by pjcast
Now, if the video starts to get behind the audio, you should drop some video frames (not render them) - it is much better than trying to drop/add audio frames (inserting blank audio or cutting audio clips out sounds really irritating, while missing a few pictures, or a paused video is not as bad).

Ok, this is what I'm trying to do now. I tried to do it by de-coupling the video rendering from the video generation engine. The results were very surprising. I'll be starting a new thread, since I think it warrants its own discussion. I'll link to it from here after I get the thread up.

Thanks!

-Joe

Ok, here it is:

Link to other thread

This is fascinating and frustrating all at the same time :)

-Joe