You have foam where waves are breaking, so that is what you want to simulate.

There exist other papers about this up to 10-15 years old, but I can't find the particular one that I have in mind (which was very good) right now...

This one however seems to describe the principle, too:
http://matthias-mueller-fischer.ch/publications/breakingWaves.pdf

The images in the paper are a bit coarse, you'd probably need 5-10 times as many particles, and have them textured in a way so there's a bit of a smooth falloff to make them look less artificial and to "multiply" them. Plus, texture the breaking wave body with a "foamy" texture on top. But in principle, that's it.