AI for audio to visual representation/displays

Are you looking at being able to do this with music? If so, what types of music? I am an ex-theory/composition major, composer, arranger, piano/keyboard player (and other instruments), producer and recording studio engineer. This is of interest to me and I may be able to help.

What exactly are you looking to be able to do with this?

You can train neural networks to recognize patterns in music. There's lots of stuff online about this. You can start here:

http://www.google.com/search?hl=en&lr=&ie=UTF-8&q=neural+network+music+pattern

Also, if you search the forum at my website, one of the regulars there, Kirk D posted a link to his paper about using ANNs to classify music. That may interest you.

Finally, someone at flipcode -- I think it was a code of the day, but not really sure -- converted some of my code so that the output from an ANN mapped onto the render area. He was getting some groovy looking stuff, so that may be of interest too.

Good luck! Sounds like a cool project.

InnocuousFox:
Yes this is to be done with music, mainly focusing on dance/pop/electronic but it should work for any genre.

I guess another way to explain it, is to automate the human process that is involved when making say a pop video or to replace the virtual VJ's at clubs (who i've rarely been impressed by)

Unfortunatley i've not been involved in music for some time, and so I'm having difficutly in finding the right words to explain my aims. Hopefully you can get the jist of what i mean though.

Basiclly if you watch any pop video they generally follow some standard practices to relate images to sound (song). This can be as simple as cutting between camera/scenes on the beat (or even half beat), having the images relate to the highs and lows of a song (e.g. breakdowns - where you lose the beat or lose the meoldy, builds where instruments are gradually layered ontop of each other over time etc).

Of course most videos these days appear to be story led or just show the band playing at a gig. What i'm looking at is much more graphic/image based. Kinda reflecting the mood of the music. This in itself is a whole addiitonal area to the the soundtolight toy. That is the actual content driven by the triggers from the sound/music anylsis. The prupose would be like the winamp plugins to entertain the user with images while listening to music.

My older projects have approached this using very graphic representations, and having the music drive mathemtaical equations to form images. It was reasonable successful, especially with our human brain so eager to see patterns in images that arn't actually there. However the draw back with this approach I found was quite often you could get the same interesting patterns just by feeding random numbers into the system and not anaylise the audio at all.

So this time i'm looking for a much tighter integration between the audio and graphic engines. For exaqmple during quiet periods have the screen almost black and empty, as the music builds up, for the screen to fill up etc.

I'm sure musical theory could help with this, not least in having better terms and descriptions for key events in a song. I'm sure there is also a knowledge on how to manipulate the listener to change their mood, or pull them inside the music. Being able to detect these and change the visual output based upon them would go along way towards my aims. In this regard I've often felt that classic dance/techno/trance etc is somewhat akin to classical music. In the way that a song can have highs and lows, and often builds on themes, meoldies throughout the piece.

Well i've started waffling now, so i'll stop. Like I said its quite hard to explain exactly what i'm after I hope this gave some clue to it.

Oh the actual audio anaylsis is not strictly for one thing, as i've got a couple of game ideas whose gameplay is to be driven by audio/music as well as the sound to light toy (virtual).

Fup:

Hmm, NN have always been a little scarey to me, never really found the time to learn about them in the depth. Then again i've not been overly impressed with previous AI methods or related methods. Genetic algorithms is one that springs to mind, always thought that equated pretty much to a brute force algorithm.

Anyway i'll check out the link, and the forum - i'd forgotten about checking out the few AI websites, being so used to using gamedev these days.

Searched your site (and read the ANN tutorial, very useful) but cold find no reference to KirkD's paper about classifying music in the forums. Any chance you have the link to hand?


thanks

[Edited by - noisecrime on July 13, 2004 12:17:13 AM]

o.k some further thoughts.

Firstly I realised just after my intial post that using the energy beat detection as the basis of the patterns to examine is flawed. Well, sort of. Its great for determining when the 'beats/high energy' of a song are, detecting drums and hit-hats effectively, but thats all it does. It obviously tells me nothing about the actual frequencies or how the song has progress from a previous timeslice.

Although the beat detection is a very important part of the algorithm, what I should be looking for is a way to describe the music overtime that can be made into a pattern. However looking at the frequency band energy moves from a 2D to 3D pattern. In the energy beat detection a cell is either on or off and with 16 in a line it creates a nice 1d image, 2D over time. With frequencies we have them across the width the energy in a band of frequencies is height, and time is depth.

For myself a move into 3D makes pattern detection harder and more complex, but I might be wrong (so many graphic/3D issues can usually be resolved by removing a dimension).

Why look at frequnecies? Well because as far as I know its the best method available currently to determine what the music is doing or rather represent the music in a more scientific way.

In a previous post I mentioned 'content', this is becuase as I see it there are two main elements to the project. The first is anaylising the audio to determine and trigger events. The second is the content that is triggered by these events. For the SoundToLight toy the content is most likely to be done via a 3D engine. Events themselves will be defined by scripts and can be either 'spot' events (something happens on a beat - camera change) or song events (the songs tempo changes - the colours of all models being display fade out) etc. These scripts can be very simple, or extremely complex.

But it doesn't stop there, something that i've always thought might be fun is to plug this into a L-system and see litteraly what 'grows' from the music. In terms of games i'm looking at songs defining the play area/level or controlling actual game events/entities.

Alas, I cannot find the link either, I'm even beginning to doubt if it was kirk who was the author of the paper!

Whatever, Kirk visits here regularly so he should be along soon to put us right.

Well, let me see if I am in the same vein as you. What I have always wanted to do (without AI/formulaic assistance) is to design what I will call "on-screen fireworks" to music. Not fireworks per se, but rather just lights and patterns that match the music - again, far better than what you see in WinAmp or MediaPlayer.

Back in my day (when I was working with a lot of bands) I would love to run the lights for them at their gigs. I was always frustrated that the light dudes (actually, "light dude" borders on an official industry title!) didn't really know/care what the music was doing. They just changed things once in a while. I wanted the colors to match the mood/intensity of the music at that moment. I wanted changes not only on the beat, but whole color/scene changes to happen on shifts in the music (e.g. verse to chorus). Having worked with those bands in the studio, I knew their music very well (often too well!) and would be able to match the lighting to their songs very well.

This is what I see being able to do with what you are talking about.

The trick is not finding beats and mimicing them. The trick is detecting changes in the music as you have alluded to. When are we changing from verse to chorus as I mentioned above? I want to SEE when the drummer changes from hi-hat to the ride cymbal! I want to SEE when the guitarist flips from a bright, airy clean channel to the heavier, distored channel.

I'm grabbing some song samples from Amazon.com to point out the changes I want to hear. Click on them to go straight to the Amazon song clip.

Nickelback: How You Remind Me
This is the end of the chorus where things would be thick and heavy. As the guitar fades, I want the remenants of that thickness to fade along with it until there is a sparse array of light/pattern left. Alone, this could be done with a simple sound/frequency detector. However, I want the computer to KNOW that we have gone back to a different section of the song and have the base coloring CHANGE right on the downbeat of that new section - leaving the color of the prior section to do the fading. I feel that the chorus is "angry colors" such as red and orange but that the verse section is more blues and purples. The only way that can happen is if the algorithm has detected that there is a significant change in the music ON THAT BEAT!

3 Doors Down: When I'm Gone
Rather the opposite this time, we go from cool to hot. Again, I want the computer to see the break. However, I want to not only detect straight beats here, but to detect the pattern of the guitar during the chorus. I want light and motion on those heavy guitar chords during the chorus. It IS helped out largely by the kick/snare work of the drummer. You automatically feel a dooowwn-UP!-dooowwn pattern between the two of them. How to represent that? How to detect it in the first place? Even better, I want those hard guitar chords to have movement in them as they are held... almost like smoke billowing but then sucked away on those breaks (along with the snare). Things like THAT I am seeing and wish I could program an algorithm to detect.

Give me examples (name only is fine) of music you are looking at.

Hi Dave,

I see this topic is going to grow ;) FIrst i'll reply to your post, then add an additional reply to where i'm at.

>> What I have always wanted to do (without AI/formulaic assistance)

Is that a typo? Its rather odd to be discussing this in the AI forum if you want to achieve your 'on-screen Fireworks'. If its not a typo could you clarrify what you mean by doing this without AI/formulaic assistance? What other methods are there?

>> Back in my day (when I was working with a lot of bands)
He,he, yeah back in my day I was a 'Rave' DJ (not that i'd usually use the term 'rave' as I don't like it, but I think its the best term to use if someone wasn't really involved in the scene so as they understand where i'm coming from). I too was frustrated with the lack of interest or understanding to construct a good light show to the music.

>> Give me examples (name only is fine) of music you are looking at.

Well really I don't have any specific songs in mind, or even specific genre's. Ideally it should work for all, or at least be tweakable to work for all.

However as i've said i'm more baised towards the dance/techno/trance scene as well as classical and soundtracks, but they all share the same thing in common and what you've indicated to in both the tracks you present. That is the song develops over time, the mood changes, it can be soaring during one part and clam the next. All the things that lighting/visual should take cues from to reflect.

Yet to be honest I don't think my interest lies with any one specific project or use for this. Instead its much more to gain a general understanding of music and 'real time' musical anyalsis so it can be applied to any situation.

In fact its imperative to gain this understanding to produce any good audio2visual application, but gaining that understanding is not going to be easy. I feel i've made some progress in terms of what I can extract from the audio data I currently have to reflect changes in the audio, but there is a long way to go.

I'd love to show you the 'piano roll' display data i've gotton from playing those amazon tracks, but currently the app doesn't support exporting them... see next post. As they look quite promising

[Edited by - noisecrime on July 14, 2004 9:59:00 AM]

Quote:

Original post by noisecrime >> What I have always wanted to do (without AI/formulaic assistance) Is that a typo? Its rather odd to be discussing this in the AI forum if you want to achieve your 'on-screen Fireworks'. If its not a typo could you clarrify what you mean by doing this without AI/formulaic assistance? What other methods are there?

Graphics programming by hand. Ya know... the old fashioned way?

Quote:

I'd love to show you the 'piano roll' display data i've gotton from playing those amazon tracks, but currently the app doesn't support exporting them... see next post. As they look quite promising

I would be interested in seeing them. Perhaps a screenshot?

First off this thread has really degenerated from a question on AI to more of a discussion and me rambling on about what i'm doing, jotting down thoughts as they come to mind. I hope that doesn't matter, and i'm sure AI will come back into it eventually. Just at this ppoint its a question of finding what to apply the eventual AI to. However i'm still interested in comments or opinions on the matter.


... So I developed my audio analysis application a bit more. Actually it feels like a gross exaggeration to call it that as its little more than a basic FFT, with a few additional algorithms and some nice displays of the data. However I have found it useful.

What I intend to do is rewrite my A2VA (Audio2Visual Analysis), making it more user friendly, with more display options, save/load features and most importantly the ability to play and track an audio file whilst displaying the real-time data.

What I mean by this is much like your traditional sound application, you can scroll along the sound wave, or play from specific spots, assign label points to jump to specific parts of the file etc. What I want and feel is needed is to tie the sound wave/song time display to the display of the FFT at the same time intervals. I hope this will allow for much easier assessment of analytical qualities, at specific points in a song.

The reason for doing this is that I’ve become quite engrossed in the ‘piano rolls’ I added, which provide a display of the changes in my data over time (I.e. its scrolling).

Currently I have two, one is the beat detection (as per the gamedev article), which amazingly I find I can now ‘read’ and can distinguish between not just a bass drum and hi-hat (quite easy) but also where there are multiple h-hats or cymbals after the initial one. This might not sound much, but I found it fascinating especially being able to ‘read’ what is basically a row of dots (pattern matching ;) )

The second is the frequency bands energy per time slice, and which I had a problem with being 3D data (see previous post). I got around the 3 dimensional aspect of the data (FFT Bands, energy and time) by converting energy value into a colour value, so instead of a cell of the piano roll being on or off it had a hue. For my purposes I just assign the energy value directly to the red component of the pixel. This has proved interesting and possibly might have a use in determine a change of mood or tone of a song. I added an additional band to this roll which is the average energy of all the bands combined, and this also appears to reflect changes in mood or tone.

Oh I should point out its not actually detecting changes in the mood or tone, but rather the energy at that time, I’m therefore assuming lots of energy means you’re listening to a ‘hot’ part of the song, whilst conversely low energy means its ‘cooled ‘ down. Not ideal I know.

During my work, its become clear to me that we need to gather as much data about the audio as possible in order to make more informed decisions. That in themselves the beat detection and FFT energy output are useful, but perhaps used in combination can help with more accurately detecting changes in a song. Hence the need for rewrite of my tool to provide better visualisation of the data, as that should help formulae the analysis strategy. Additional data combined with existing data could hopefully refine the apps competency.

However there is a problem…..

Hey Dave,

o.k I quickly added the abilty to output the piano roll or rather a section of it to a file. A little photoshop later (for combining the beat and the FFT rolls and resizing) I have 3 sections for you to look at.

However I must stress this is early development, that events are very subtle to identify, and it would help alot to actual see the direct relationship between the display and the song as it plays.

Each sample shows approx 2 bars worth of data and contains two grids. Each grid shows time along the x-axis, and FFT bands on the Y-axis. (FFT bands refers to progressive banding). There are 16 bands in total, but i foget the overall freq range these encompass.

The top grid is the beat detection (in blue) where each cell (FFT band) is on or off indicating if the energy level has trigged a beat. The second is the FFT Band Energy levels (red) the strength of the colour indicating the enrgy level. The top row of the FFT Energy grid represents the average energy of all the bands combined.


Start: This is the very start of the piece on amazon.
Section Yeah

Break: This is where the song breaks down and the guitar comes in as you pointed to in your post.
Section Break

When viewing these make sure they are at full size (1024 wide)

Things to note.
On the start piece if you look very carefully you'll see the FFT energy levels rise up before the beats start. There is then a high frequency of FFT energy during this 'hot' point of the track.

Moving onto the 'yeah' sample, that has reduced freq of high FFT energy as its not quite as 'busy' a section, but the average energy remains relatively high.

In the final sample, the 'break' the freq of high FFT energy reduces again as can be seen (although barely) on the average enrgy levels. In addition the beat detection looks a little more 'pronounce' on this sample, bit more 'messy' on the others.

Like I said all very subtle and difficult to see on this exmaple, but other kinds of music can provide better results. Really the best way of seeing this data is live, and like i said i think there may be btter ways of using the data to get more accurate results.

Would it be helpful if I could give you some frequency ranges for kick/snare/hat so that you could pull those out of the clutter of vocals and instrumental stuff?

You guys seem to be carrying this discussion well between you and making progress, so I don't want to detract from that progress. I can offer up some ideas though, based on the fact that I currently spend most of my days doing complex mathematical analysis of nonlinear signals, looking for ways to characterise such signals and identify certain events within them. I might, therefore, be able to help with choosing and applying some analysis methods that extend beyond the basic frequency methods you are using.

I've read through the above, so I believe I have the jist of what you are presently doing with your beat/frequency analysis... but I want to clarify exactly what it is you want from the signal, rather than what you want to use it for.

Are you trying to identify events at certain frequencies, or more abstractly, changes from one regime of the music to another. I presume it's actually the latter, in that you want to find partitions in the music 'space' that, when crossed, trigger a graphical transition as well. Is this roughly correct?

I've got a few ideas for analysis that I'll play around with, using the sound samples that Dave posted above (unless there are some others that you would rather I look at). Once I better understand what you want from the signal, I should be able to suggest some methods for getting it and provide some mathematics for it, if needed.

Cheers,

Timkin

One of the assets that you need to tap into is musical mathematics. If you can look for patterns such as beats per measure and measures per phrase, you can significantly narrow down your search space. Obviously this works well in predictable music such as dance and house music. However, when working with progressive rock, for example, the non-standard time signatures (and changing time sigs) would be far more difficult to ascertain.

Dave:
Sure somefreq ranges might be useful, however i'm doubtful i could pull those out of the 'clutter'. At present i can simply get an FFT of the frequencies, so if other instruments or vocals share the same freq as kick/hat/snare then I can't differentiate between them. I think thats getting into a whole other are of audio analysis and not something i've come across yet.

As to musical mathematics that sounds interesting, and agree that ascertaining a change in time signiture would be pretty hard.

Timkin:
Wel if it wasn't for Dave, I think i'd be left here mubbling to myself. So i welcome any other input, even though as the thread has progressed i've realised I was proberbly asking the wrong question at the start.

Clarification is a little harder as I feel I'm still exploring this whole area, and so at the moment would like to achieve everything ;) I think really i've gone back a step, to actually look at what data I have and what can be used from that data to make educated guess on what is happening to the music at that point or for trigger points.

However i stil feel pretty limited by the data I do have, (amplitude, FFT, FFT Enrgy beat detection) so any other data that can be extracted from an audio source is bound to prove useful - if you have any ideas.

Just a couple of quick questions...

For your frequency analysis, are you computing a STFT (Short Time Fourier Transform), or just a single transform over a block of data? I presume it's actually the first but with non-overlapping windows of data of a fixed length. What amount of data (in seconds) are you currently performing your analysis over, for a single 'time slice'?

Timkin

Ah, well as i've alluded to (possibly in a message i've yet to post - got a few of those), i've not been responsable for the FFT as its a pre-exisitng lib that i'm using. Truth be told I mainly work in MM DIrector, although do some C++ and openGL, not to mention writing actual C++ plugins (xtras) for Director.

So instead of all the headache of learning how to write FFT's (which frankly could be outside my math abilities - haven't checked in a while) i've used one by Antoine Schmitt, and it can be found here FFT Xtra

From the read me

The way the asFFT Xtra works :
getFFT(object me, float amplMax, int normalized, int equalized)

it continuously records incoming sound from the current sound input into two buffers of length 4*timesmooth*binCount.
At all times, one of the buffers is being used for recording, the other one (the previous one) is used for FFT computations.
When the application program requests the FFT values, the Xtra computes 'timesmooth' FFTs over the previous buffer, applies a Bartlett Window to the FFT values, averages the 'timesmooth' values for each bin and computes the power of these values.
It then applies a log function and a multiplication factor to these values so that the values are in the range of maxAmpl.


I then call this function every 1/60th of a second, although i've yet to decide what a good call frequency should be.

What i've added ontop is part of the beat detection method in the GD.net Article. I'm very keen to start the re-write of the app, but client work is as ever getting in the way, maybe i'll get to it over the weekend.

As i've relasied I need to do far more research I spent a bit of time over on AI-junkie reading the ANN and SOM tutorials, they look interesting. Was wondering wether SOM might be useful for analysising the data, as it appears to organise data including relationships in a 2D image/pattern. Perhaps this can be used over time to help determine more factors about the audio, i don't know - needs to be tested.

Like I said above, i'd been writing a few addiitonal replies, but using Word (due to my awful spelling). They just discuss some of the issues and methods i'm looking at.

So going back to where I said
'However there is a problem…..'


I’ve remembered one of the biggest issues facing this sort of application and that is I want it to be ‘real time’ Now in terms of the algorithms that’s fine it needs a fair bit of cpu but nothing modern PC’s can’t handle. No, the issue is unlike a lighting director, or video producer the application has no prior knowledge of the music or song. Yet this is something that is pretty pivotal for a human in order to be good and effective at their job. Of course humans can ‘wing’ it but in terms of real-time I’d be surprised if an application could even match that.

This is the fundamental issue facing a real-time application, lack of fore-knowledge of the song. If the song could be analysed first then an application could do a very good job at producing a competent lightshow or audio2visual display. I guess with the speed of computers these days, it should be possible to analysis say a 5 minute song in much less time, but ultimately that’s not where I want to go.

This means I can’t see an application really being able to respond ‘on a beat’ as Dave described for the Nickelback example. The reason being there is nothing on any particular beat that you could or should use for determine the change in direction for the song, a beat is just a beat. Instead you’d need to be looking towards at least 2 or 4 bars ahead (or in my case cos its real-time, behind).

There is an obvious solution to the issue of predication or lack of fore-knowledge (which is considerable) and that is to use a delay on the music. By this I mean that the audio output first goes through the PC, which buffers it for x amount of time/bars before routing it out to the speakers. This delay would allow the PC to be ahead of the audio output giving it plenty of time change its display on cue.

Sadly though I’m nowhere near being able to do that, as I’m working at a higher level, using libraries for most things (like the initial FFT). However if the results of the audio analysis work out it would definitely be the logical conclusion to develop the software towards. Of course its totally impractical for Dj’s or bands as they like to hear the music as they play it, not on a delay ;)

Yet the overall issue still stands., which is essentially that whilst it might be possible to develop a application that can achieve all that has been discussed, the chances of it reacting at the right time in realtime is pretty low.

--- More ramblings.

Up till now all I’ve been talking about is musical analysis, which is really only half the problem, the other half being the visual display, or more precisely ‘what to display’. Clearly a narrow ranged pre-defined list of ‘effects isn‘t going to cut it, so any visualisation application will need to have a large library of them. This is a challenge in itself.


However this still doesn’t address what effects should happen for specific events. For that I’m guessing some type of scriptable (XML?) files will be best. Before we can approach that though we need a definition of events and what form of effect should be associated with them. This is not as straight forward as it sounds, not least because (for me) its hard to define in words specific events. Also I’m wondering at what point something like an ANN should be introduced.


For example

EFFECTS
An effect is a script initiated upon an event. It can consist of ‘sub-effects’ (e.g. reference another effect) and is updated every frame. The basis for any effect is a link to one or more Elements as it is these that are displayed and the effect simply modifiers them based on events.

ELEMENTS
In my case because I intend this to be done in 3D an element is any 3D object. The object itself consists of geometry and shader/texture data in order for it to be displayed.


EVENT LIST - CUES

BEAT - possible effect list
Colour swap
Element swap
Element create/destroy


ENERGY DROP (e.g song fades out to string section) possible effect list
Colour fade
Display fade (transparency)
Number of Elements (Elements start disappearing )


ENERGY RISE (e.g song fades in with strings) possible effect list
Colour fade
Display fade (transparency)
Number of Elements (Elements start appearing )