Sign in to follow this  

C++ Frequency Modulation Synthesis Algorithms

This topic is 827 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

G'day mates,
 
To preface, if you are unfamiliar with frequency modulation synthesis, have a quick skim of this Wikipedia article: https://en.wikipedia.org/wiki/Frequency_modulation_synthesis
 
I have made this thread because I am trying to solve a difficult programming problem. Forgive me for not hastily getting to the point/my question; there is quite a lot of assumed knowledge of both C++ and audio synthesis in general that will be required to help me with my problem, so I have written a detailed post covering the basics of frequency modulation before delving into the autistic guts of my problem. Even if you are unable to provide me with any assistance, I hope you enjoy reading the thread (because it was fun to write).
 
Finally and before I begin, if anyone is interested in audio synthesis and video game console sound chips in general, have a look at this fun post I made some time ago on the TIGSource forums: http://forums.tigsource.com/index.php?topic=28653.msg795584#msg795584
 
Anyway, I am working on a real-time audio synthesis library in C++ styled upon sound chips of the 16-bit video game console era (Sega Genesis/Mega Drive, Neo Geo, etc.). I have a preference for the audio synthesis philosophy and methodology of Yamaha, who are a major producer of sound chips and electronic keyboards, most of which make use of frequency modulation synthesis.
 
The primary key to understanding frequency modulation synthesis is the difference between carrier operators and modulation operators. An FM operator is in its simplest form an oscillator which in most cases generate a sine wave, although my synthesizer defines 19 different types of wave forms (which is more or less irrelevant in the present context) and implements the ability to receive prerecorded wave data as well. An FM operator can be either a modulator, carrier or both. A modulator is fed into other operators and adjusts their frequency regardless of whether or not the latter is a carrier or another modulator. Carrier operators are not fed into other operators, but output the resulting wave data. Multiple carriers may be blended together to create the final output, usually by simply averaging them together.
 
This diagram conveys the difference between carrier operators and modulation operators:
 
3zNbW6z.gif
 
FM synthesizers require at least 2 operators, one of which must be a carrier. A single operator (which is inevitably a carrier) cannot realistically be considered as frequency modulated synthesis because it is not modulated by anything (unless it is being modulated by prerecorded wave data instead of another operator, although that is besides the point).
 
The secondary key to understanding frequency modulation synthesis the concept of 'algorithms'. An FM algorithm defines the way that multiple operators (carriers and modulators) interact with one another. The number of operators that a particular FM synthesizer makes use of (which has in the past been constantly defined depending on the design of the synthesizer or limitations of the hardware) sets an upper limit to the number of possible algorithms. Similarly and historically, the number of algorithms and their respective configurations have been constantly defined.
 
The audio synthesis library I am developing differs from previous generations of FM synthesizers in that the number of operators, number of algorithms and algorithm configurations are not constantly defined or restricted by hardware limitations (i.e. they are defined by the user).
 
The majority of video game console FM sound chips make use of 4 operators. Notably, the Yamaha YM2612 sound chip (famously incorporated into the Sega Genesis/Mega Drive) defines 8 algorithms for its 4 operators, which are displayed here:
 
WuAtnqY.png
 
Although the YM2612 defines only 8 algorithms, many more are possible. There is an upper limit to the number of possible algorithms based on 4 operators, but I have not determined what it is, and that is part of the problem I am trying to solve.
 
More advanced FM synthesizers (such as electronic keyboards and virtual instruments) often make use of more than 4 operators. The number of possible algorithms to an FM synthesizer increases exponentially with each additional operator. Typically, the designers of these FM synthesizers define a constant number of possible algorithms and fixed algorithm configurations based on a trial and error test case methodology to determine which algorithms output the most interesting and useful audio. That being said, the upper limit of possible algorithm configurations in the case of higher numbers of operators can defy imagination.
 
To convey exactly how mind-boggling the possibilities are, here is a small subset of predefined algorithms to an electronic keyboard utilizing 6 operators:
 
i9hIpKs.png

Now that I have established the fundamentals of frequency modulation synthesis, we can move on to the specifics of my code.

I have yet to determine a method of mapping the possible algorithms based on the user defined number of operators in my synthesizer. I have already attempted to manually define all possible algorithms in the range of 1-5 operators, yet being a mere human being makes me prone to error. There are far too many variables to take into consideration, and with every additional operator I attempt to solve, another dimension is opened that I am not able to comprehend.

In the meantime, I have declared a function pointer towards a function that calculates the output of a single frame iteration as so:
 
f32 (voice::*iterate_func)(f32**,s32,f32);

When the user attempts to change the algorithm number (which I have attempted to predefine, yet missed the mark), this function is executed:
 
void voice::set_algorithm( s32 v )
{
	BDAPI_MUTEX( mutex );

	switch( operator_amount )
	{
		case 1:
		{
			algorithm = 0;

			iterate_func = &voice::iterate_1_0;

			return;
		}
		case 2:
		{
			switch( algorithm = v % 2 )
			{
				case 0: iterate_func = &voice::iterate_2_0; return;
				case 1: iterate_func = &voice::iterate_2_1; return;
			}			
		}
// etc

These iteration functions invoke the operators as modulators and carriers based upon their function parameters; an operator iteration that receives the output from another operator is modulated:
 
f32 voice::iterate_1_0( f32** o, s32 i, f32 p_0 )
{
	return o[0][i] = operators[0]->iterate( 0.f, p_0 );
}
f32 voice::iterate_2_0( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(          );
	o[1][i] = operators[1]->iterate( 0.f, p_0 );
	
	return ( o[0][i] + o[1][i] ) / 2.f;
}
f32 voice::iterate_2_1( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate();

	return o[1][i] = operators[1]->iterate( o[0][i], p_0 );
}

// etc

Now, when I crank the number of operators into overdrive (5 in this particular case) things quickly become hectic and confusing:
 
f32 voice::iterate_04_00( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate( o[1][i] );
	o[3][i] = operators[3]->iterate( o[2][i] );

	return o[4][i] = operators[4]->iterate( o[3][i], p_0 );
}
f32 voice::iterate_04_01( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate( o[1][i] );
	o[3][i] = operators[3]->iterate(         );

	return o[4][i] = operators[4]->iterate( ( o[2][i] + o[3][i] ) / 2.f, p_0 );
}
f32 voice::iterate_04_02( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate(         );
	o[3][i] = operators[3]->iterate( o[2][i] );

	return o[4][i] = operators[4]->iterate( ( o[1][i] + o[3][i] ) / 2.f, p_0 );
}
f32 voice::iterate_04_03( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate(         );
	o[3][i] = operators[3]->iterate(         );

	return o[4][i] = operators[4]->iterate( ( o[1][i] + o[2][i] + o[3][i] ) / 3.f, p_0 );
}
f32 voice::iterate_04_04( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate();
	o[1][i] = operators[1]->iterate();
	o[2][i] = operators[2]->iterate();
	o[3][i] = operators[3]->iterate();

	return o[4][i] = operators[4]->iterate( ( o[0][i] + o[1][i] + o[2][i] + o[3][i] ) / 4.f, p_0 );
}
f32 voice::iterate_04_05( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate();
	o[1][i] = operators[1]->iterate();
	o[2][i] = operators[2]->iterate();
	o[3][i] = operators[3]->iterate( ( o[0][i] + o[1][i] + o[2][i] ) / 3.f );

	return o[4][i] = operators[4]->iterate( o[3][i], p_0 );
}
f32 voice::iterate_04_06( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(                             );
	o[1][i] = operators[1]->iterate(                             );
	o[2][i] = operators[2]->iterate( ( o[0][i] + o[1][i] ) / 2.f );
	o[3][i] = operators[3]->iterate( o[2][i]                     );

	return o[4][i] = operators[4]->iterate( o[3][i], p_0 );
}

// etc

My failure is that I am unable to account for all possible algorithm configurations when the number of operators is variable. Manually coding each and every algorithm I can think of up to 5 operators seems impossible, and I don't even want to begin thinking about the implications of 6 operators. However, many advanced FM synthesizers that have existed for decades (electronic keyboards, etc.) have made use of 6 operators and beyond. My aim is to create the most thorough and flexible frequency modulated synthesis system thus far and I feel that there is a way to harness the power of C++ to do so (which has so far eluded me).

The code I have presented to you above will most likely be scrapped, but I think it does a sufficient job of conveying the need for some sort of C++ template metaprogramming or perhaps a simpler and more straight forward method. Performance is critical, so I want to minimize overhead as much as possible.

For reference, here is the code for a few of the functions specificed above:
 
// iterate
f32 operate::iterate( f32 modulation, f32 pulse )
{
	BDAPI_MUTEX( mutex );

	operator_oscillator->set_frequency( interpolate_frequency() );

	return last_frame = operator_oscillator->iterate( powf( 2.f, pulse ) * powf( 2.f, modulation )
		* ( 1.f + multiplier ) ) * envelope->iterate() * velocity;
}

// private interpolate frequency
f32 operate::interpolate_frequency()
{
	if( envelope->get_state() != adsr::attack )
	{
		return target_frequency;
	}
	else
	{
		return math::mix<f32>( previous_frequency, math::clamp<f32>( math::inv_lerp<f32>( last_adsr,
			envelope->get_last_frame(), envelope->get_attack_level() ), 1.f ), target_frequency );
	}
}

// run
raw& voice::run()
{
	BDAPI_MUTEX( mutex );

	if( voice_state == idle )
	{
		return *output;
	}

	fori( samples )
	{
		if( using_amp )
		{
			if( using_pulse )
			{
				output_data[i] = ( this->*iterate_func )( operator_data, i, lfo_pulse->iterate() )
					* 1.f - ( lfo_amp->iterate() + 1.f ) * 0.5f;
			}
			else
			{
				lfo_pulse->skip();

				output_data[i] = ( this->*iterate_func )( operator_data, i, 0.f )
					* 1.f - ( lfo_amp->iterate() + 1.f ) * 0.5f;
			}
		}
		else
		{
			lfo_amp->skip();

			if( using_pulse )
			{
				output_data[i] = ( this->*iterate_func )( operator_data, i, lfo_pulse->iterate() );
			}
			else
			{
				lfo_pulse->skip();

				output_data[i] = ( this->*iterate_func )( operator_data, i, 0.f );
			}
		}
	}

	if( voice_state == state::key_on && duration > 0.f )
	{
		duration -= samples;

		if( duration <= 0.f )
		{
			fori( operator_amount )
			{
				operators[i]->key_off();
			}

			voice_state = state::key_off;
		}
	}

	if( voice_state == state::key_off )
	{
		bl set_idle = true;

		fori( operator_amount )
		{
			operators[i]->update_state();

			if( operators[i]->get_state() == idle )
			{
				operators[i]->clear();
			}
			else
			{
				set_idle = false;
			}
		}

		if( set_idle )
		{
			voice_state = idle;

			output->clear();
		}
	}

	return *output;
}

Any ideas whatsoever would be greatly appreciated.

Thank you for your patience.

C6ZU6IK.jpg Edited by sigvatr

Share this post


Link to post
Share on other sites

If I understood correctly, you want to algorithmically define (or find) all possible combinations of your operators (few of them seen in 2nd and 3rd pictures)?

 

In that case I have these questions:

  1. in 2nd image (e), (f), (g), (h) have multiple outputs, is that correct?
  2. in 3rd image doesn't seem to have an arrow dedicated to output, so where exactly it is?
  3. in 3rd image (1), (2), (3), etc. there is a line that goes from and into same box, are those loops? If yes, how many times it is looped?

I'm not sure whether it provides solution you want, but you might look into random graph generation (possibly directed graph).

That being said, 5 operators will be equivalent to 5 vertices in a graph. Complete graph with 5 vertices contains up to 10 edges. Edge can either exist or not exist, that makes only 2 states. Therefore number of all graphs is 2^10 = 1024. On a side note, 6 vertices is 15 edges, so you're in for a nice ride here.

However if graph is directed it can go forwards, backwards, both directions, neither direction. That makes 4^10 = 1,048,576 total possibilities.

If you have only single output then you can randomly choose one of vertices (thus x5 possibilities). But if you want multiple outputs then you are presented with 2^5 possibilities (different output order will be accomplished through different graphs).

Using this method you will encounter graphs where certain operators do not contribute to the final output (isolated vertices for example), thus you can throw them away, which results in a subset of your operators.

 

I hope this information will help you.

Share this post


Link to post
Share on other sites

in 2nd image (e), (f), (g), (h) have multiple outputs, is that correct?


No, they are added/averaged together to return a single output.

in 3rd image doesn't seem to have an arrow dedicated to output, so where exactly it is?


It appears that the operators run from top to bottom.

in 3rd image (1), (2), (3), etc. there is a line that goes from and into same box, are those loops? If yes, how many times it is looped?


I'm not sure exactly how this particular synthesizer is working. Either these are modulators that modulate themselves (which I think is unlikely) or they are generating feedback.

That makes 4^10 = 1,048,576 total possibilities.


Yes, I was expecting something like this to be the case. I don't think I want to manually implement each one of those.

Share this post


Link to post
Share on other sites

To confirm, your thing wont basically have those loops in it? So its a directed acyclic graph (DAG)? (If a node can feedback to itself and itself only, that would should not make the problem any harder, just 1 bit for each node feedback/no feedback)

 

And every node which only has outputs, is a carrier and those will be combined to obtain final output.

 

So you need some kind of algorithm to iterate over all possible DAGs given N vertices, (each vertex being an operator), and find the ones with no outputs and those are the carriers.

 

 

Now, do you want to generate the resulting algorithms at compile time or at runtime?

 

If at compile time, c++ metaprogramming (and/or macros?) can do that (I think its turing complete), but it totally was not designed for such tasks, and you need to be a wizard to do it.

So generating the algorithms at runtime might be a better idea (and doesnt force picking a finite amount of algorithms at compile time). I dont know how your thing works, but if possible, generate them once at the beginning, and reuse that data, instead of regenerating each time one is needed.

 

EDIT:

Though if you wanted to do this compile time, it would only make sense for a small amount of operators. So I assume you want to do it at runtime, and if you dont want to, you eventually will ;)

 

I assume you dont need to actually iterate over possible DAGs (I dont see the use in that), but instead want the ability to generate random DAGs deterministically from a seed (so they may be regenerated later), which google gives some answers for.

 

For performance, you can then take the output of that and either make your users write it as compile time algorithm by hand, or write some script to generate the C/C++ code for it (seems easier than templates IMO).

Edited by Waterlimon

Share this post


Link to post
Share on other sites

Yes, I was expecting something like this to be the case. I don't think I want to manually implement each one of those.


I solved this problem in my FM synthesizer (small blurb) by using a sequence of instructions dictating how the operators were chained. For instance:
/*
program:
	u          unit to render
	in         phase buffer input slot, 0 for null input
	out        phase buffer output slot, 0 for audio data
*/
//   ????
//   ????? ????
//   ?? 0??? 1???
//    ???? ???? ? ????
//              ??? 3??? out
//         ???? ? ????
//         ? 2???
//         ????
idx = 0; //          u  in out
i.program[idx++] = { 0, 0, 2 };
i.program[idx++] = { 1, 2, 1 };
i.program[idx++] = { 2, 0, 1 };
i.program[idx++] = { 3, 1, 0 };
i.ct_program = idx;
The operators were then processed in series:
fill phase_buffer, 0
for I in instructions:
  U = @units[I.u]
  ? = (U.freq_mul*@note.freq + U.freq_offset)*two_pi / @sample_rate
  in_buffer = phase_buffer + @ct_samples*I.in
  out_buffer = phase_buffer + @ct_samples*I.out

  for sdx in 0...@ct_samples:
    out_buffer[sdx] += U.amp * (sin U.initial_phase + ?*sdx + in_buffer[sdx])

copy phase_buffer, @out_audio_data, ct_samples
If you want to constrain your FM synthesizer like the OPN chips, then perhaps you could limit your synthesizer to a set of 8 programmable algorithms, and then have your units index into that. Edited by fastcall22

Share this post


Link to post
Share on other sites

This topic is 827 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this