The new scheme eliminates this squarish blockiness. The tiled approach blending has been replaced with a vertex-by-vertex approach (as I should have done initially). In this new method, each vertex is assigned an alpha value for each of the terrain layers. This alpha value is stored in 8 separate arrays or VBOs, and is passed to the terrain-rendering shader as a vertex attribute. The shader replaces the alpha for the final color with this attribute. (This lets me upload vertex and normal data to a VBO once, then never touch it again, and I can change the alpha map just by switching buffer sources.) The end result is terrain transitions that are vastly more pleasing to the eye, smoother, and not restricted to large tilish blocks. I can also manipulate the alpha values to sharpen or blur terrain transitions as desired. All in all, it's a large improvement, and it only took a few very minor modifications to the code.
Here is an example of the final result. Note that this shot does not manipulate the alphas to smooth the blend between types; each vertex is assigned either a 1 or a 0 for a given layer's alpha. Much smoother transitions are possible. In this first shot, the terrain is flattened, in the second it is elevated to see how it would look in-game.
A very trivial change, but I think the result is well worth it. I have in mind as well a few ideas to further optimize it, even though it already runs pretty fast.