G'day mates,

To preface, if you are unfamiliar with frequency modulation synthesis, have a quick skim of this Wikipedia article: https://en.wikipedia.org/wiki/Frequency_modulation_synthesis

I have made this thread because I am trying to solve a difficult programming problem. Forgive me for not hastily getting to the point/my question; there is quite a lot of assumed knowledge of both C++ and audio synthesis in general that will be required to help me with my problem, so I have written a detailed post covering the basics of frequency modulation before delving into the autistic guts of my problem. Even if you are unable to provide me with any assistance, I hope you enjoy reading the thread (because it was fun to write).

Finally and before I begin, if anyone is interested in audio synthesis and video game console sound chips in general, have a look at this fun post I made some time ago on the TIGSource forums: http://forums.tigsource.com/index.php?topic=28653.msg795584#msg795584

Anyway, I am working on a real-time audio synthesis library in C++ styled upon sound chips of the 16-bit video game console era (Sega Genesis/Mega Drive, Neo Geo, etc.). I have a preference for the audio synthesis philosophy and methodology of Yamaha, who are a major producer of sound chips and electronic keyboards, most of which make use of frequency modulation synthesis.

The primary key to understanding frequency modulation synthesis is the difference between carrier operators and modulation operators. An FM operator is in its simplest form an oscillator which in most cases generate a sine wave, although my synthesizer defines 19 different types of wave forms (which is more or less irrelevant in the present context) and implements the ability to receive prerecorded wave data as well. An FM operator can be either a modulator, carrier or both. A modulator is fed into other operators and adjusts their frequency regardless of whether or not the latter is a carrier or another modulator. Carrier operators are not fed into other operators, but output the resulting wave data. Multiple carriers may be blended together to create the final output, usually by simply averaging them together.

This diagram conveys the difference between carrier operators and modulation operators:



FM synthesizers require at least 2 operators, one of which must be a carrier. A single operator (which is inevitably a carrier) cannot realistically be considered as frequency modulated synthesis because it is not modulated by anything (unless it is being modulated by prerecorded wave data instead of another operator, although that is besides the point).

The secondary key to understanding frequency modulation synthesis the concept of 'algorithms'. An FM algorithm defines the way that multiple operators (carriers and modulators) interact with one another. The number of operators that a particular FM synthesizer makes use of (which has in the past been constantly defined depending on the design of the synthesizer or limitations of the hardware) sets an upper limit to the number of possible algorithms. Similarly and historically, the number of algorithms and their respective configurations have been constantly defined.

The audio synthesis library I am developing differs from previous generations of FM synthesizers in that the number of operators, number of algorithms and algorithm configurations are not constantly defined or restricted by hardware limitations (i.e. they are defined by the user).

The majority of video game console FM sound chips make use of 4 operators. Notably, the Yamaha YM2612 sound chip (famously incorporated into the Sega Genesis/Mega Drive) defines 8 algorithms for its 4 operators, which are displayed here:



Although the YM2612 defines only 8 algorithms, many more are possible. There is an upper limit to the number of possible algorithms based on 4 operators, but I have not determined what it is, and that is part of the problem I am trying to solve.

More advanced FM synthesizers (such as electronic keyboards and virtual instruments) often make use of more than 4 operators. The number of possible algorithms to an FM synthesizer increases exponentially with each additional operator. Typically, the designers of these FM synthesizers define a constant number of possible algorithms and fixed algorithm configurations based on a trial and error test case methodology to determine which algorithms output the most interesting and useful audio. That being said, the upper limit of possible algorithm configurations in the case of higher numbers of operators can defy imagination.

To convey exactly how mind-boggling the possibilities are, here is a small subset of predefined algorithms to an electronic keyboard utilizing 6 operators:



Now that I have established the fundamentals of frequency modulation synthesis, we can move on to the specifics of my code.

I have yet to determine a method of mapping the possible algorithms based on the user defined number of operators in my synthesizer. I have already attempted to manually define all possible algorithms in the range of 1-5 operators, yet being a mere human being makes me prone to error. There are far too many variables to take into consideration, and with every additional operator I attempt to solve, another dimension is opened that I am not able to comprehend.

In the meantime, I have declared a function pointer towards a function that calculates the output of a single frame iteration as so:

f32 (voice::*iterate_func)(f32**,s32,f32);

When the user attempts to change the algorithm number (which I have attempted to predefine, yet missed the mark), this function is executed:

void voice::set_algorithm( s32 v )
{
	BDAPI_MUTEX( mutex );

	switch( operator_amount )
	{
		case 1:
		{
			algorithm = 0;

			iterate_func = &voice::iterate_1_0;

			return;
		}
		case 2:
		{
			switch( algorithm = v % 2 )
			{
				case 0: iterate_func = &voice::iterate_2_0; return;
				case 1: iterate_func = &voice::iterate_2_1; return;
			}			
		}
// etc

These iteration functions invoke the operators as modulators and carriers based upon their function parameters; an operator iteration that receives the output from another operator is modulated:

f32 voice::iterate_1_0( f32** o, s32 i, f32 p_0 )
{
	return o[0][i] = operators[0]->iterate( 0.f, p_0 );
}
f32 voice::iterate_2_0( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(          );
	o[1][i] = operators[1]->iterate( 0.f, p_0 );
	
	return ( o[0][i] + o[1][i] ) / 2.f;
}
f32 voice::iterate_2_1( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate();

	return o[1][i] = operators[1]->iterate( o[0][i], p_0 );
}

// etc

Now, when I crank the number of operators into overdrive (5 in this particular case) things quickly become hectic and confusing:

f32 voice::iterate_04_00( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate( o[1][i] );
	o[3][i] = operators[3]->iterate( o[2][i] );

	return o[4][i] = operators[4]->iterate( o[3][i], p_0 );
}
f32 voice::iterate_04_01( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate( o[1][i] );
	o[3][i] = operators[3]->iterate(         );

	return o[4][i] = operators[4]->iterate( ( o[2][i] + o[3][i] ) / 2.f, p_0 );
}
f32 voice::iterate_04_02( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate(         );
	o[3][i] = operators[3]->iterate( o[2][i] );

	return o[4][i] = operators[4]->iterate( ( o[1][i] + o[3][i] ) / 2.f, p_0 );
}
f32 voice::iterate_04_03( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(         );
	o[1][i] = operators[1]->iterate( o[0][i] );
	o[2][i] = operators[2]->iterate(         );
	o[3][i] = operators[3]->iterate(         );

	return o[4][i] = operators[4]->iterate( ( o[1][i] + o[2][i] + o[3][i] ) / 3.f, p_0 );
}
f32 voice::iterate_04_04( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate();
	o[1][i] = operators[1]->iterate();
	o[2][i] = operators[2]->iterate();
	o[3][i] = operators[3]->iterate();

	return o[4][i] = operators[4]->iterate( ( o[0][i] + o[1][i] + o[2][i] + o[3][i] ) / 4.f, p_0 );
}
f32 voice::iterate_04_05( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate();
	o[1][i] = operators[1]->iterate();
	o[2][i] = operators[2]->iterate();
	o[3][i] = operators[3]->iterate( ( o[0][i] + o[1][i] + o[2][i] ) / 3.f );

	return o[4][i] = operators[4]->iterate( o[3][i], p_0 );
}
f32 voice::iterate_04_06( f32** o, s32 i, f32 p_0 )
{
	o[0][i] = operators[0]->iterate(                             );
	o[1][i] = operators[1]->iterate(                             );
	o[2][i] = operators[2]->iterate( ( o[0][i] + o[1][i] ) / 2.f );
	o[3][i] = operators[3]->iterate( o[2][i]                     );

	return o[4][i] = operators[4]->iterate( o[3][i], p_0 );
}

// etc

My failure is that I am unable to account for all possible algorithm configurations when the number of operators is variable. Manually coding each and every algorithm I can think of up to 5 operators seems impossible, and I don't even want to begin thinking about the implications of 6 operators. However, many advanced FM synthesizers that have existed for decades (electronic keyboards, etc.) have made use of 6 operators and beyond. My aim is to create the most thorough and flexible frequency modulated synthesis system thus far and I feel that there is a way to harness the power of C++ to do so (which has so far eluded me).

The code I have presented to you above will most likely be scrapped, but I think it does a sufficient job of conveying the need for some sort of C++ template metaprogramming or perhaps a simpler and more straight forward method. Performance is critical, so I want to minimize overhead as much as possible.

For reference, here is the code for a few of the functions specificed above:

// iterate
f32 operate::iterate( f32 modulation, f32 pulse )
{
	BDAPI_MUTEX( mutex );

	operator_oscillator->set_frequency( interpolate_frequency() );

	return last_frame = operator_oscillator->iterate( powf( 2.f, pulse ) * powf( 2.f, modulation )
		* ( 1.f + multiplier ) ) * envelope->iterate() * velocity;
}

// private interpolate frequency
f32 operate::interpolate_frequency()
{
	if( envelope->get_state() != adsr::attack )
	{
		return target_frequency;
	}
	else
	{
		return math::mix<f32>( previous_frequency, math::clamp<f32>( math::inv_lerp<f32>( last_adsr,
			envelope->get_last_frame(), envelope->get_attack_level() ), 1.f ), target_frequency );
	}
}

// run
raw& voice::run()
{
	BDAPI_MUTEX( mutex );

	if( voice_state == idle )
	{
		return *output;
	}

	fori( samples )
	{
		if( using_amp )
		{
			if( using_pulse )
			{
				output_data[i] = ( this->*iterate_func )( operator_data, i, lfo_pulse->iterate() )
					* 1.f - ( lfo_amp->iterate() + 1.f ) * 0.5f;
			}
			else
			{
				lfo_pulse->skip();

				output_data[i] = ( this->*iterate_func )( operator_data, i, 0.f )
					* 1.f - ( lfo_amp->iterate() + 1.f ) * 0.5f;
			}
		}
		else
		{
			lfo_amp->skip();

			if( using_pulse )
			{
				output_data[i] = ( this->*iterate_func )( operator_data, i, lfo_pulse->iterate() );
			}
			else
			{
				lfo_pulse->skip();

				output_data[i] = ( this->*iterate_func )( operator_data, i, 0.f );
			}
		}
	}

	if( voice_state == state::key_on && duration > 0.f )
	{
		duration -= samples;

		if( duration <= 0.f )
		{
			fori( operator_amount )
			{
				operators[i]->key_off();
			}

			voice_state = state::key_off;
		}
	}

	if( voice_state == state::key_off )
	{
		bl set_idle = true;

		fori( operator_amount )
		{
			operators[i]->update_state();

			if( operators[i]->get_state() == idle )
			{
				operators[i]->clear();
			}
			else
			{
				set_idle = false;
			}
		}

		if( set_idle )
		{
			voice_state = idle;

			output->clear();
		}
	}

	return *output;
}

Any ideas whatsoever would be greatly appreciated.

Thank you for your patience.

If I understood correctly, you want to algorithmically define (or find) all possible combinations of your operators (few of them seen in 2nd and 3rd pictures)?

In that case I have these questions:

1. in 2nd image (e), (f), (g), (h) have multiple outputs, is that correct?
2. in 3rd image doesn't seem to have an arrow dedicated to output, so where exactly it is?
3. in 3rd image (1), (2), (3), etc. there is a line that goes from and into same box, are those loops? If yes, how many times it is looped?

I'm not sure whether it provides solution you want, but you might look into random graph generation (possibly directed graph).

That being said, 5 operators will be equivalent to 5 vertices in a graph. Complete graph with 5 vertices contains up to 10 edges. Edge can either exist or not exist, that makes only 2 states. Therefore number of all graphs is 2^10 = 1024. On a side note, 6 vertices is 15 edges, so you're in for a nice ride here.

However if graph is directed it can go forwards, backwards, both directions, neither direction. That makes 4^10 = 1,048,576 total possibilities.

If you have only single output then you can randomly choose one of vertices (thus x5 possibilities). But if you want multiple outputs then you are presented with 2^5 possibilities (different output order will be accomplished through different graphs).

Using this method you will encounter graphs where certain operators do not contribute to the final output (isolated vertices for example), thus you can throw them away, which results in a subset of your operators.

I hope this information will help you.

in 2nd image (e), (f), (g), (h) have multiple outputs, is that correct?

No, they are added/averaged together to return a single output.

in 3rd image doesn't seem to have an arrow dedicated to output, so where exactly it is?

It appears that the operators run from top to bottom.

in 3rd image (1), (2), (3), etc. there is a line that goes from and into same box, are those loops? If yes, how many times it is looped?

I'm not sure exactly how this particular synthesizer is working. Either these are modulators that modulate themselves (which I think is unlikely) or they are generating feedback.

That makes 4^10 = 1,048,576 total possibilities.

Yes, I was expecting something like this to be the case. I don't think I want to manually implement each one of those.

Yes, I was expecting something like this to be the case. I don't think I want to manually implement each one of those.

I solved this problem in my FM synthesizer (small blurb) by using a sequence of instructions dictating how the operators were chained. For instance:

/*
program:
	u          unit to render
	in         phase buffer input slot, 0 for null input
	out        phase buffer output slot, 0 for audio data
*/
//   ????
//   ????? ????
//   ?? 0??? 1???
//    ???? ???? ? ????
//              ??? 3??? out
//         ???? ? ????
//         ? 2???
//         ????
idx = 0; //          u  in out
i.program[idx++] = { 0, 0, 2 };
i.program[idx++] = { 1, 2, 1 };
i.program[idx++] = { 2, 0, 1 };
i.program[idx++] = { 3, 1, 0 };
i.ct_program = idx;

The operators were then processed in series:

fill phase_buffer, 0
for I in instructions:
  U = @units[I.u]
  ? = (U.freq_mul*@note.freq + U.freq_offset)*two_pi / @sample_rate
  in_buffer = phase_buffer + @ct_samples*I.in
  out_buffer = phase_buffer + @ct_samples*I.out

  for sdx in 0...@ct_samples:
    out_buffer[sdx] += U.amp * (sin U.initial_phase + ?*sdx + in_buffer[sdx])

copy phase_buffer, @out_audio_data, ct_samples

If you want to constrain your FM synthesizer like the OPN chips, then perhaps you could limit your synthesizer to a set of 8 programmable algorithms, and then have your units index into that.

Now, do you want to generate the resulting algorithms at compile time or at runtime?

Runtime.