What I did on one of my very first projects was to make a wavemanager. What I basically did was make a class that consisted all the information I needed about a wave, like how many enemies are alive or what number of wave this was.

The class basically had 2 functions I used:

A spawnwave function which just added 10 new enemies (hardcoded, but you can play with it obviously). In my case I called this function when all 10 enemies were dead or off the screen.

I also had a tick function which only handles the enemy logic.

The wavemanager was an object of a level object, so I could just call that function whenever a wave was done. You can use whatever rule you want to spawn a new wave of enemies and even keep track of how many waves have already been so you can stop spawning waves at a certain point.

I could post the source, but it's in C++, let me know if you want to see it anyway and I'll try to refactor it, it was one of my first projects and it's ugly

It's not much of an algorithm really, but here goes! (Note that this is old and in my opinion ugly!)

Wavemanager.cpp
void WaveManager::SpawnWave()
{
MobsWaveAlive = 10;
Waves++;
for(i = 0; i < 10; i++)
{
MobsWave = new Enemy();
MobsWave->currentEnemyType = Enemy::EnemyType::RANGED;
MobsWave->init();
VecEnemyPosWave = MobsWave->getEnemyPos();
VecEnemyPosWave.x = SCRWIDTH + VecEnemyPosWave.x - (i * 48) ;
MobsWave->setEnemyPos(VecEnemyPosWave);
VelocityX = 1.0f;
VelocityY = 0;
MobsWave->SetMobVx(VelocityX);
if ( Waves % 2)
{
MobsWave = new Enemy();
MobsWave->currentEnemyType = Enemy::EnemyType::MELEE;
MobsWave->init();
VecEnemyPosWave = MobsWave->getEnemyPos();
VecEnemyPosWave.x = SCRWIDTH + VecEnemyPosWave.x - (i * 48) ;
MobsWave->setEnemyPos(VecEnemyPosWave);
VelocityX = 1.0f;
VelocityY = 0;
MobsWave->SetMobVx(VelocityX);
}


}
}

Everytime I call the spawnwave function. I increment wave by 1 to keep track of how many waves.
I set the amount of enemies that are alive to 10. I had a fixed number of enemies, but you can diverse that with some tuning.

void WaveManager::Tick(Surface* m_screen, float a_DT)
{
for(int i = 0; i < 10; i++)
{
if(MobsWave != NULL)
{
MobsWave->tick(m_screen, a_DT);
MobsWave->doLogic(a_DT);
}
}
}

For every enemy (the fixed 10) I just update their whatever logic they have, all handled in the enemy class itself.

level.cpp
if (!isEnd)
{
if (wavemanager->Waves == 2)
powerup->tick(m_Screen, a_DT);
wavemanager->Tick(m_Screen, a_DT);
for (int i = 0; i < 10; i ++)
{
Enemy* enemy;
Spell* spell;
Spell* enemyspell;
for (int j = 0; j < 1; j++)
{
if (wavemanager->MobsWave != NULL)
{
enemyspell = wavemanager->MobsWave->m_spell[j]->getSpell();
if (collision->Intersect(enemyspell, player))
{
player->setIsAlive(false);
wavemanager->MobsWave->m_spell[j]->setIsAlive(false);
}
}
}
if (wavemanager->MobsWave != NULL)
{
spell = player->m_spell->getSpell();
enemy = wavemanager->MobsWave->GetEnemy();
if (collision->Intersect(powerup, player))
{
player->m_spell->CurrentPower = Spell::Type::UPGRADE;
powerup->setIsAlive(false);
}
if (collision->Intersect(enemy, player))
{
player->setIsAlive(false);
}
if (player->m_spell->getIsAlive())
{
if (collision->Intersect(enemy, spell))
{
player->m_spell->setIsAlive(false);
wavemanager->MobsWave->removeEnemyHealth(player->m_spell->getSpellDamage());
if (wavemanager->MobsWave->getEnemyHealth() <= 0)
{
wavemanager->MobsWave->setIsAlive(false);
SAFE_DELETE(wavemanager->MobsWave);
int nummobs = GAME->getLevel()->getWaveManager()->GetMobsAlive() -1;
GAME->getLevel()->getWaveManager()->SetMobsAlive(nummobs);
if (nummobs <= 0)
{
GAME->getLevel()->getWaveManager()->SpawnWave();
}
}
//SAFE_DELETE(player->m_spell);
}
}
}
}
}

I created the wavemanager in my level class, I only had one level so it didn't matter in my case. In this part I handled the collision testing with the player and the enemy and their spells. If an enemy got killed, I decremented the counter in my wavemanager and if that reached zero, I spawned another wave.

Now with the keeping track of how many waves have spawned you can decide to use that as a factor of spawning a new wave, of you could use a certain amount of time between each wave. Because I used a fixed amount for every wave and I only spawned a new wave when all the enemies were dead, you will most likely need to refactor this to suite your need more, but the general idea behind this should be clear