Jump to content
  • entries
    12
  • comments
    6
  • views
    1376

Working with sound when creating rhythm game (TERRORHYTHM project)

EvilCG

943 views

The first question every rhythm-game developer asks himself: “Should my gameplay actually be powered by music?” If the answer is yes and it’s not enough for the developer to have just visual scene feedback to playing music, the second question appears: “Should I give the player possibility to upload his own track to play”? Sure, you can have the same question in the first case too, but only if your gameplay is really affected by music the answer is crucial for you. Because giving player ability to upload his music means to spend resources on developing custom music analyzer and level generator and also on wide genre testing and tuning. Delegate-level creation to Steam Workshop community doesn’t seem like a good idea till you don’t have the community itself.

5a3bc560c028e_main_menu.thumb.JPG.4e21254f4f1c6a4767d3bd9a6a2ea010.JPG

We understood that program sound analyzer is a rather popular task, though still not trivial and trying to solve it with our small studio is the responsible decision. And we still could not resist adding this killer-feature, allow a player to crush souls to the rhythm of his favorite song. 

Rhythm is center of our gameplay. So-called BPM(beats per minute) with its frequency you move your head to cool track. But if you want to build your game around the rhythm you should get it first. Music for the game engine is only set of samples he can play and some information about them.

BPM extraction is not a new task - even if you ignore rhythm games, DJ equipment manufacturers and digital audio workstations developers have done a great job in this area. And still after all these achievements they often delegate rhythm detection to human. 

Screenshot_2.jpg.ff9e9427a1c18f01cfe293fae9ba487a.jpg

- tap CUE button to the beat and CDJ will catch it

We reviewed a lot of materials before came up with the solution that worked for us.
Music is complex soundwave, the vibration of air in your room, in the club or between your headphones membrane and eardrum. Transformation of these vibrations to something melodic and rhythmic (what our brain does automatically) requires some calculations. Most part of these calculations is Fast Fourier Transform that converts the complex sound signal to the sum of sinusoids with certain frequencies and amplitudes. For those who can’t understand in what way curve from school connected to sound we will explain: the sound wave of sinusoidal form is the simplest timbre familiar to anyone. Phone line ringing for example. The frequency of this sine is sound pitch and amplitude is volume. Here are some FFT for different sound waveforms for illustrative purposes:

Screenshot_6.jpg.8a03a1040d7df5ae3555a31a13b16e8f.jpg

Of course for music resulting frequency spectrum will be much more complex - all over the range it will have some frequencies. If you remember any music spectrum analyzer (people often call it wrong: equalizer) you will understand what we are talking about. This is the clear picture of what sound look after Fast Fourier Transform.

Screenshot_3.jpg.609c4491c0e153dea1425b34a33829d7.jpg

 

- the loudness of frequencies from lowest to highest (left to right)

Thanks to the game engine that made this transform for us we can work with music like with the array of samples (number of these samples per second depends on quality and compression) that gives us information about all the sounding frequencies at every moment of the record. 

So we have several tasks to solve with help of this data.

First and main task - BPM extraction. Almost every in-game parameters and values are linked to the rhythm - from enemies spawn and move speed to animations playback speed and combo behavior. 

To extract rhythm we should first understand what it is. Usually, by beat we mean start point of the short repeated musical piece. Looping of this piece determines main song tempo. Classic rhythm-section consists of drums and bass (whether it rock or jazz, metal or dance music). This knowledge gives us approximate borders of the frequency range that we will scout for volume peaks. Our main goal is under 120Hz range, but besides it, we will also scan additional narrow range around 1kHz where often rhythm-section-supporters sit.

First, we should understand loudness range for these frequencies along all the song. These values are needed to make relative decisions. When we got minimum and maximum volume we can start the analysis. In short, this analysis is a iterating over samples from song beginning to its end and comparison of needed frequencies amplitude with min and max values and with siblings. By making kind of “low-frequency volume jumps map” we are trying to find periodical sequences - peaks, repeating with the same time period (luckily for music lovers and unluckily for music analyzers, bass and drum parts can be complex and different). 
Most frequent value of this time period we are using as beat duration, which is used to calculate BPM.

5a3bbe4abfc60_Screenshot_6ENG.jpg.41e2fd365b40e5ba5b9e481c2af27912.jpg

Analyzer reacts different to various genres - of course, house music, where the clean beat is marked by kickdrum, is the simplest one to analyze. Dealing with aggressive metal genres, where the drummer can do incredibly fast and furious things, is much harder. In that case, we can ask additional frequency range for help. Searching for snare and electric guitar drops can help in beat determination. We should also remember about music genres with floating bar size, where musical phrases can be different size and duration - jazz, math rock and others.

Sound compression is one more thing that can hinder analysis. This way of post-processing music is used by many producers nowadays. Compression is all about narrowing difference between the loudest and the quietest frequencies (so-called “dynamic range”) through the song to make it sound louder without quality loss and extreme peaks. Obviously, extreme compression (which can often be heard in dance music) can be a barrier to drawing proper “low-frequency volume jumps map” and make analysis results wrong.

Screenshot_7ENG.jpg.6ea3268a4df42f0564e81cffc0652340.jpg

Bad overall record quality can be the problem too - big amount of noises at the wide range or bad mixing can cause frequency conflicts (garbage side frequencies emergence).

To provide some support for beat extraction algorithm we add sync during the playback. The pre-analyzed track gives us average BPM value. According to this value, we set beat duration and run main game timer. Then during song playback, we periodically synchronizing this timer to our “map of peaks”. If deviation is big, next synchronization is set to short period of time and gameplay in this period gets softer to protect player from system errors.

Add some small features to this analysis and we get working version of beat-sync mechanism. Now we can concentrate on interactive levels - spawn more difficult enemies sequences when music is intense, visual scene feedback, etc.

Anyway, to avoid the bad experience for a player who wants to play the game to old grandpa vinyl or arrhythmic metal of his friends' band, we honestly say after the attempt to upload song - sorry we failed to get this track tempo, please find something else.

Game Steam page - TERRORHYTHM



1 Comment


Recommended Comments

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement
  • Advertisement
  • Blog Entries

  • Similar Content

    • By trapazza
      I'm trying to add some details like grass, rocks, trees, etc. to my little procedurally-generated planet. The meshes for the terrain are created from a spherified cube which is split in chunks (chunked LOD).
      To do this I've wrote a geometry shader that takes a mesh as input and uses its vertex positions as locations where the patches of grass will be placed (as textured quads).
      For an infinite flat world (not spherical) I'd use the terrain mesh as input to the geometry shader, but I've found that this won't work well on a sphere, since the vertex density is not homogeneous across the surface.
      So the main question would be: How to create a point cloud for each terrain chunk whose points were equally distributed across the chunk?
      Note: I've seen some examples where these points are calculated from intersecting a massive rain of totally random perpendicular rays from above... but I found this solution overkill, to say the least.
      Another related question would be: Is there something better/faster than the geometry shader approach, maybe using compute shaders and instancing?
    • By FedGuard
      Hello all,
       
      I would like to start off with thanking you all for this community. Without fora like these to assist people the already hard journey to making an own game would be exponentially more difficult. Next I would like to apologize for the long post, in advance...
      I am contemplating making a game. There, now that's out of the way, maybe some further details might be handy.
      I am not some youngster (no offence) with dreams of breaking into the industry, I am 38, have a full-time job, a wife, kid and dog so I think I am not even considered indie? However I recently found myself with additional time on my hands and decided I would try my hand at making a game.Why? Well mostly because I would like to contribute something, also because I think I have a project worth making (and of course some extra income wouldn't hurt either to be honest). The first thing I realized was, I have absolutely no relevant skill or experience. Hmm; ok, never mind, we can overcome that, right?
      I have spent a few months "researching",meaning looking at YouTube channels, reading articles and fora. Needless to say, I am more confused now than when I started. I also bought some courses (Blender, Unity, C#) and set out to make my ideas more concrete.
      I quickly discovered, I am definitely not an artist... So I decided, though I do plan to continue learning the art side eventually, I would focus on the design and development phase first. The idea being, if it takes me a year or more solely learning stuff and taking courses without actually working on my game, I would become demoralized and the risk of quitting would increase.
      So I thought I would:
      1: Keep following the courses Unity and C# while starting on the actual game development as the courses and my knowledge progress.
      2: Acquire some artwork to help me get a connection with the game and main character, and have something to helm keep me motivated. (I already did some contacting and realized this will not be cheap...). Also try to have the main character model so I can use it to start testing the initial character and game mechanics. For this I have my first concrete question. I already learned that outsourcing this will easily run up in the high hundreds or thousands of dollars... (lowest offer so far being 220 USD) I am therefore playing with the idea of purchasing https://assetstore.unity.com/packages/3d/animations/medieval-animations-mega-pack-12141 with the intention of then have an artist alter and/or add to the animations (it is for a Roman character so some shield animations are not going to work the same way.). This way I could start  with the basic character mechanics. Is this a good idea, waste of money,...? Any suggestions? I then have a related but separate question. Is it a good idea to buy Playmaker (or some other similar software I haven't yet heard of like RPGAIO), and using this for initial build, then changing/adding code as the need arises?
      3.Get a playable initial level ready as a rough demo and then starting to look for artist for level design and character/prop creation.
      ...
       
      I would really appreciate some input from more experienced people, and especially answers to my questions. Of course any advice is extremely welcome.
    • By william.equal
      Hey, I just finished a new episode of "Game Audio Lookout"! This time it's about musical sound effects in the Super Mario series. Here's the link to the video on YouTube:
      --
      Musical Sound Effects in the Super Mario Series | Game Audio Lookout
      https://www.youtube.com/watch?v=6hHbTVloizU
      We’ll have a deeper look at musical sound effects in the Super Mario series in this episode of "Game Audio Lookout".
      I guess everybody has heard the sounds of the Super Mario series before. But I believe most of us don’t exactly know how these were constructed and what efforts were taken in later instalments of the series to produce sound effects that even harmonise with the game’s music.
      --
      Feel free to let me know what you think
      Alex
    • By GameTop
      Dirt Bike Extreme - another game made with Unity. Took about 2 months to complete.
      Take part in extreme motorcycle races across the dangerous and challenging tracks. Dirt Bike Extreme is easy to pick up but hard to master. Race, jump and crash your way and other mad rivals through the amazing tracks as you master the skills and physics of motocross in this high-speed racing adventure. Conquer challenging routes on 23 different runs, discover new bikes and become the best of the best! Over 257K downloads already!
      Windows Version:
      https://www.gametop.com/download-free-games/dirt-bike-extreme/

      Mac Version:
      https://www.macstop.com/games/dirt-bike-extreme/
       

       


    • By Jordan Winslow
      Hey guys, my name is Jordan Winslow and I am a professional electronic music producer & composer who also happens to be a talented story writer and has a ton of experience with VNMaker, Tyrannobuilder, Renpy, and RPG Maker tools and I am looking for talented artists who want to make a horror game together!
      Last Horror Project I Composed Music For: 
       
      My last game I created: "The Watchers"  https://jordanwinslow.itch.io/the-watchers
      My Music: https://jordanwinslow.me/showcase
      I am open to plot ideas but, based on your artwork, I would like to create an original story that matches our music and art so we can play on both of our strengths. Up until now I have only been able to create visual novels with the use of stock photography and stock videos, but with the use of original art, we should be able to come up with a story that is far more specific to the art on screen.
      My favorite horror games (For reference)
      Saya no Uta (Horror Visual Novel Originally in Japanese)
      The Crooked Man 
      SOMA
      Corpse Party
       
      I have a few pretty awesome ideas for sci-fi or extra-dimensional horror games, I also know of a great real-life story about aliens we could make a game about and I could easily come up with something new if I am inspired by your art!
       
      So let's create something awesome together! Send me a message along with some examples of your art and we will either create a team of multiple people together or just the two of us if you are capable of coming up with a decent amount of original artwork! I've got the music, the story and the programming handled unless you want to do the programming and help with the story.
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!