I've experimented with all of the above, and currently use a combination of it, to varying degrees. It mostly depends on the textures used, some 'thin' vegetation such as typical grass will fade out quicker due to the darker average of the alpha channel, whereas thick foliage for tree leafs will become more opaque in the smaller mipmaps.

Alpha Blending
-smooth, no jaggies
-requires depth sorting
-lose parallax effect without depth-testing
-trouble with other transparency effect (particles, screen-space effect such as DOF, SSAO)
-overall not worth the trouble

Alpha Testing
-tweak alpha cutoff value to match texture
-keep texture alpha anti-aliased for smooth edges (use DXT5)

Mipmaps
-sharpen mipmap alpha
-or use point-sampling rather than box/average
-or tweaked by artists
-DXT1 (1 bit alpha) will make mipmapping at distance thicker, but up close jaggy edges
-use distance-fields as Andy mentioned, Valve has a nice paper on this
-no anti-aliasing

Alpha-to-coverage
-nicely anti-aliased
-loses alpha density at distance due to mipmapping
-scaling alpha * 2 usually fixes this, but makes foliage up close more dense and edges slightly jaggy (can compensate by tweaking alpha)
-alternatively use a distance falloff in shader to compensate

And to illustrate:



Btw, shade the leafs by computing the vertex normal from the center of the bushy parts like this (I call this 'veggie-normals'):

-construct bounding box of all leaf polygons
-construct a line segment that runs in the center along the longest dimension
-for each leaf polygon vertex, compute the closest point on this line
-create a vector from this point to the vertex, normalize it
-store this normal

And in your shader, do regular dot3 lighting. This setup will give you nice smooth lighting across the foliage. If you want specular/bump, you can do this also but you'll also need to store the regular vertex normals. Store veggie-normals in the RGB of 4-byte vertex colors, you can use the alpha for other interesting purposes, such as a weight for wind animation etc.

Indeed some good visuals. I'd also like to have an alpha-blended image for comparision, if it doesn't take you all to much to produce one. Because that's what we're doing at the moment, and it looks just fine for my taste - without depth sorting, but with depth buffer writes active and with alpha-testing >0.1 in addition.

Has anyone attempted to apply Valve's Alpha-tested Magnification technique to foliage rendering? It offers beautiful results for alpha-testing on text, and might work pretty well for leaves and billboard trees.

@swiftcoder, I did brief attempt, but the result was a bit disappointing for dense foliage (such as in the above screens). But I think it should work fairly well for grass and 'thinner' foliage. Of course the downside is that you need to do the art (or at least the alpha channel) at 8x or more resolution. 2x or 4x wasn't enough in my tests to make a noticeable difference.

@Schrompf: why, sure:



Note that this is with depth-testing, without the glDepthMask trick, so it has some ugly edges (reduced by alpha-test with very low cutoff value). We render the skybox after all major geometry, so it would completely overwrite anything above the horizon.
You can see that quality wise, it is close to alpha-to-coverage. Of course, less flexible.
For low-end hardware and non-multisampled framebuffer we fall back to regular alpha-testing. This actually makes non-anti-aliased images look more consistent.

@B_old: thanks, nope it was created by me and one of our artists. cgtextures.com and mayang is a good place to start if you're looking for free high-quality textures.

On another note: at some point we had alpha-tested vegetation, with only 4 mipmaps (512x512) to keep the textures sharp in the distance. However, we got complaints about this being 'noisy'. It may also be bad for performance on low-end systems, mipmaps can reduce texture caching traffic since in ideal situations only the mipmap is transfered to cache rather than the entire texture.