Advertisement Jump to content
Sign in to follow this  

Transforming 3d objects/scene by music???

This topic is 4322 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi. For some time now, I´m looking for tutorials/papers/links which describe effects which transform a scene or make a scene(3d scene) or 3d objects reacting on the played music. A common problem is, I have not the correct or any good search keywords for google. What do I need to produce such effects? Effects you have seen in certain demos, or in all demos!? Grateful for all good suggestions!!!! Thanks Alex

Share this post

Link to post
Share on other sites
Assuming that you aren't interested in scripting the events according to predetermined music, you'll need to get intimate with a little signal processing.

First, you should set up a real-time audio capture. I know that DirectSound and FMOD are capable of this. Ideally, you'd have a looped capture buffer that, at any moment, contains the raw audio data over the last 100ms or so.

If you're only concerned with loudness and potentially 'business' (that's busy-ness), then you can get away with analysing this as it stands. However, if you want your program to be anything but tone-deaf, you'll need to transform the data into the frequency domain. There are a whole bunch of transforms that can help you out here.

If you'd like to be able to poll any frequency for its mean amplitude, then the Fast Fourier Transform (FFT) is for you. There are dozens of FFT libraries out there (of varying quality) and the transform itself is well-documented.

If, however, you're more concerned with notes (in the musical sense) than frequencies, then you should look into the Q-transform. This will give you periodic 'tone-aware' data so you can poll for the mean amplitude of any given note.

There's a lot more to learn, but I think this should get you started.


Share this post

Link to post
Share on other sites
Hi Admiral.

Thanks. Signal processing. I encountered some places on the web and some books while I was searching for "image processing". The university-lib nearby has some good books about that topic.
Everything you wrote is absolutly new to me! I never have spend time learning/reading something about those topics you mentioned.

But, maybe you have some links to resources, because, you know, its hard to learn the theory without any aim!!!
With resources I mean something like(sorry for beeing so demanding) papers from the demoscene where those guys describe a bit how they have done such things.
Many sites are just down, a others site, from the groups themselves, have no information...

Grateful for response.


[Edited by - directNoob on March 22, 2007 6:03:02 PM]

Share this post

Link to post
Share on other sites
What you're getting into is an entire new world. Fortunately for you, you won't need to know too much about it to do some of the things roughly resembling what you want to do.

For what you want to do, all you need is to convert the audio into frequency domain (for quick reference: using FFT - Google this) and analyze the spectrum that you get. You'll have a range of frequencies from 0 to N Hertz which you can then track to respond to certain cues.

However - for instance you'll have a hard time responding to speech (since this requires you to do real-time speech recognition in one form or another), but you'll be able to respond to cues according to their magnitude and position in the spectra of the incoming signals. Eg, it's not that difficult to differentiate a bass beat from the rest of the signal - you just monitor frequencies below 150 Hz for peaks.

Mind you, however, that responding to a signal like this can be very tricky because without having timbral references, you won't be able to tell if a peak is eg a sharp s-sound as said by a female voice or one of the harmonics of a high-pitched violin along the lines of the shower scene in Psycho.

To conclude, you're treading on water that is both very vast and very deep and all anyone can do is wish you luck with coping with it.

The definitive point to start at is The Scientist and Engineer's Guide to Digital Signal Processing, avialble for free to download at the link. It's a book, 640 pages in length, simple to read and it'll open a lot of (but not all) doors for you.

Good luck.

Share this post

Link to post
Share on other sites
Thanks alot!

Ok, now, this is interesting. All topics you talked about are absolutly new and undiscovered to me! How can I say. Before some time in the past, I thought that I'll be satisfied with 2D and 3D programming with effects. But I just saw the toe of that big monster. This is a really big field of artistic and mathmatical properties, which overwhelmed me hard. I was shocked! Books over books according this topics.
I also don't know what I have expected!?!?!

In my opinion, today it is more hard to start such undertakings, than in the past, because of this huge and big and complex super thing. And without any school experiance like university experiance, it is, in my opinion, absolutly not possible to cope those things!!!!!! I realized it just after the first semester of applied science in computing.

But anyway, thanks for your help, again!

I will start reading very soon and will see if I can do what I want to do, namely , moving thing with nice music and some good sounds... ;)

But with your help, now I have a really good starting point.


Here is such a nice example what I mean:
There is this "Variform remixed" topic, which is a (demo)music player,
which alters lights and/or graphics and/or scene by the music.
The download is 93 Mb big. But it is worth it, really!!!
And by the way, kewlers are doing great demos!

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using, you agree to our community Guidelines, Terms of Use, and Privacy Policy. is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!