Jump to content
  • Advertisement
  • entries
    12
  • comments
    6
  • views
    1411

Working with sound when creating rhythm game (TERRORHYTHM project)

EvilCG

985 views

The first question every rhythm-game developer asks himself: “Should my gameplay actually be powered by music?” If the answer is yes and it’s not enough for the developer to have just visual scene feedback to playing music, the second question appears: “Should I give the player possibility to upload his own track to play”? Sure, you can have the same question in the first case too, but only if your gameplay is really affected by music the answer is crucial for you. Because giving player ability to upload his music means to spend resources on developing custom music analyzer and level generator and also on wide genre testing and tuning. Delegate-level creation to Steam Workshop community doesn’t seem like a good idea till you don’t have the community itself.

5a3bc560c028e_main_menu.thumb.JPG.4e21254f4f1c6a4767d3bd9a6a2ea010.JPG

We understood that program sound analyzer is a rather popular task, though still not trivial and trying to solve it with our small studio is the responsible decision. And we still could not resist adding this killer-feature, allow a player to crush souls to the rhythm of his favorite song. 

Rhythm is center of our gameplay. So-called BPM(beats per minute) with its frequency you move your head to cool track. But if you want to build your game around the rhythm you should get it first. Music for the game engine is only set of samples he can play and some information about them.

BPM extraction is not a new task - even if you ignore rhythm games, DJ equipment manufacturers and digital audio workstations developers have done a great job in this area. And still after all these achievements they often delegate rhythm detection to human. 

Screenshot_2.jpg.ff9e9427a1c18f01cfe293fae9ba487a.jpg

- tap CUE button to the beat and CDJ will catch it

We reviewed a lot of materials before came up with the solution that worked for us.
Music is complex soundwave, the vibration of air in your room, in the club or between your headphones membrane and eardrum. Transformation of these vibrations to something melodic and rhythmic (what our brain does automatically) requires some calculations. Most part of these calculations is Fast Fourier Transform that converts the complex sound signal to the sum of sinusoids with certain frequencies and amplitudes. For those who can’t understand in what way curve from school connected to sound we will explain: the sound wave of sinusoidal form is the simplest timbre familiar to anyone. Phone line ringing for example. The frequency of this sine is sound pitch and amplitude is volume. Here are some FFT for different sound waveforms for illustrative purposes:

Screenshot_6.jpg.8a03a1040d7df5ae3555a31a13b16e8f.jpg

Of course for music resulting frequency spectrum will be much more complex - all over the range it will have some frequencies. If you remember any music spectrum analyzer (people often call it wrong: equalizer) you will understand what we are talking about. This is the clear picture of what sound look after Fast Fourier Transform.

Screenshot_3.jpg.609c4491c0e153dea1425b34a33829d7.jpg

 

- the loudness of frequencies from lowest to highest (left to right)

Thanks to the game engine that made this transform for us we can work with music like with the array of samples (number of these samples per second depends on quality and compression) that gives us information about all the sounding frequencies at every moment of the record. 

So we have several tasks to solve with help of this data.

First and main task - BPM extraction. Almost every in-game parameters and values are linked to the rhythm - from enemies spawn and move speed to animations playback speed and combo behavior. 

To extract rhythm we should first understand what it is. Usually, by beat we mean start point of the short repeated musical piece. Looping of this piece determines main song tempo. Classic rhythm-section consists of drums and bass (whether it rock or jazz, metal or dance music). This knowledge gives us approximate borders of the frequency range that we will scout for volume peaks. Our main goal is under 120Hz range, but besides it, we will also scan additional narrow range around 1kHz where often rhythm-section-supporters sit.

First, we should understand loudness range for these frequencies along all the song. These values are needed to make relative decisions. When we got minimum and maximum volume we can start the analysis. In short, this analysis is a iterating over samples from song beginning to its end and comparison of needed frequencies amplitude with min and max values and with siblings. By making kind of “low-frequency volume jumps map” we are trying to find periodical sequences - peaks, repeating with the same time period (luckily for music lovers and unluckily for music analyzers, bass and drum parts can be complex and different). 
Most frequent value of this time period we are using as beat duration, which is used to calculate BPM.

5a3bbe4abfc60_Screenshot_6ENG.jpg.41e2fd365b40e5ba5b9e481c2af27912.jpg

Analyzer reacts different to various genres - of course, house music, where the clean beat is marked by kickdrum, is the simplest one to analyze. Dealing with aggressive metal genres, where the drummer can do incredibly fast and furious things, is much harder. In that case, we can ask additional frequency range for help. Searching for snare and electric guitar drops can help in beat determination. We should also remember about music genres with floating bar size, where musical phrases can be different size and duration - jazz, math rock and others.

Sound compression is one more thing that can hinder analysis. This way of post-processing music is used by many producers nowadays. Compression is all about narrowing difference between the loudest and the quietest frequencies (so-called “dynamic range”) through the song to make it sound louder without quality loss and extreme peaks. Obviously, extreme compression (which can often be heard in dance music) can be a barrier to drawing proper “low-frequency volume jumps map” and make analysis results wrong.

Screenshot_7ENG.jpg.6ea3268a4df42f0564e81cffc0652340.jpg

Bad overall record quality can be the problem too - big amount of noises at the wide range or bad mixing can cause frequency conflicts (garbage side frequencies emergence).

To provide some support for beat extraction algorithm we add sync during the playback. The pre-analyzed track gives us average BPM value. According to this value, we set beat duration and run main game timer. Then during song playback, we periodically synchronizing this timer to our “map of peaks”. If deviation is big, next synchronization is set to short period of time and gameplay in this period gets softer to protect player from system errors.

Add some small features to this analysis and we get working version of beat-sync mechanism. Now we can concentrate on interactive levels - spawn more difficult enemies sequences when music is intense, visual scene feedback, etc.

Anyway, to avoid the bad experience for a player who wants to play the game to old grandpa vinyl or arrhythmic metal of his friends' band, we honestly say after the attempt to upload song - sorry we failed to get this track tempo, please find something else.

Game Steam page - TERRORHYTHM



1 Comment


Recommended Comments

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement
  • Advertisement
  • Blog Entries

  • Similar Content

    • By Tristanb4
      I've been making music for about 7 years, I have hundreds of releases on soundcloud and bandcamp. Recently I have stepped up my post production game, pouring long hours into EQ and mixing. Most of my music is in a moody, "foggy" piano style with heavy experimentation through pitch shifting, overdubbing, and live recording. I use a spectrogram EQ to manually shape sounds and scoop out noise in Audacity. I am familiar with many general concepts, applying compression, reverb, high and low pass filters, and pretty much all of the effects in Audacity and many of the pitfalls and lessons of live recording for guitar and piano in my home studio. I am familiar with some other programs like ableton and fruity loops but live recording is my strong suit as opposed to composing music in a DAW. I rely heavily on improvisation, recording large amounts of audio and cutting it down and manipulating it in post as well as doing overdubs. I can put out a project of piano music in a month or so up to what I think is a high / acceptable standard that I personally am happy with.

      I am heavily inspired by Akira Yamaoka's work on the Silent Hill series as well as Angelo Badalamenti. I dream of composing music for games or short films, and feel like I'm ready to take on a project like that, as well as being willing to license my already existing music out which I think would be a perfect fit for the right type of horror game or anything with emotional elements.

      I am currently working on another project that will be released in December or on New Years. I will work for a reasonable amount and have done this out of passion for 5+ years because I love doing it. I feel that I have improved enough now to pursue doing something like this.

      Thank you so much to anyone who even bothers to click any of these links, and thank you for your time!

      Here are my links, and you can also email me directly at tristan.best@gmail.com

      www.soundcloud.com/domonemesis
      https://tristanb.bandcamp.com/
      https://www.facebook.com/TristanBMusic
      https://twitter.com/tbest253


      Other skills: I do all of my own cover art with digital photo editing and subsequently also have about 5 years of experience with that- photography and digital photo manipulation. I can work on marketing materials or art in this way. I play the Piano, Guitar, Synth / String piano etc, and I sing. I have close connections to some other musicians and visual artists. I will be honest if I don't think my music will work for your project or if I'm not sure if I can do something well enough, but I feel comfortable taking on some general audio design as well, including general sound / dialogue recording or noise reduction.
    • By tspartant
      Hey everyone! My name is Ryan and I am the founder of Visualistic Studios, LLC. 
      I'm looking for experienced developers interested in short term and long term contracting. For the past 3 years I've been working in game development contracting, and the past year I've been working full time from home. Since then, I've received more and more contracts and I'm now at the point that I have too many for myself to handle. I have at least another full time job's worth of programming offers, and around 30-80 hours a month for 3D modeling/animation. I also am in need of a UI artist, so if you're talented please contact me! I have contracts using Unity as well as Unreal Engine 4, so if you can program in either please contact me. 
      If you are interested in working on these contracts, please send me links to your work and you hourly rate. (Most contracts range be between 18-25$/h, but please provide your normal hourly rate)
      You can get ahold of me through email - "ryan.hobbs@visualisticstudios.com", or Discord "TSpartanT#4670"
       
      Thank you everyone for reading, hope to hear from you soon!
    • By MiTi
      Dear everyone, this is my newest game, please check out and give me feedback. Thanks for your consideration.

      Overview: Cross n Puzz is a creative and addicting word puzzle game. It not only challenges your brain but also improve memory and other types of cognitive function.

      For IOS: https://itunes.apple.com/app/crossword-puzzle-image/id1435575074

      For Android: https://play.google.com/store/apps/details?id=com.caag.crosswordnpuzzle

      Game trailer: https://www.youtube.com/watch?v=stNuktpJH44&feature=youtu.be
      Crossword Puzzle Image Trailer Official.mp4  
    • By KevDev
      A Unity Asset for customizable lowpoly weapons and shields.
      - Full Version: http://u3d.as/1hvG
      - Free Version: http://u3d.as/1jZV
      You can get the character model used in the video here (for FREE!): http://u3d.as/1kP7
       

      View full story
    • By KevDev
      A Unity Asset for customizable lowpoly weapons and shields.
      - Full Version: http://u3d.as/1hvG
      - Free Version: http://u3d.as/1jZV
      You can get the character model used in the video here (for FREE!): http://u3d.as/1kP7
       
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!