So with blending you have two colors: SrcColor, which is the color output by your pixel shader, and DstColor, which is the color that's currently in the render target. The equation goes like this:
FinalColor.rgb = SrcColor.rgb * SrcBlend [BlendOp] DstColor.rgb * DstBlend;
FinalColor.a = SrcColor.a * SrcBlendAlpha [BlendOpAlpha] DstColor.a * DstBlendAlpha;
So with the settings you've specified, you're doing this:
FinalColor.rgb = SrcColor.rgb * SrcColor.rgb + DstColor.rgb * (1.0f - DstColor.rgb);
FinalColor.a = SrcColor.a * 0 + DstColor.a * 0;
This is clearly not what you want. Instead you want to multiply the SrcColor by your source alpha value, and your DstColor by the inverse of your source alpha:
FinalColor.rgb = SrcColor.rgb * SrcColor.a + DstColor.rgb * (1.0f - DstColor.a);
You can achieve this by setting SrcBlend = SRC_ALPHA, and DestBlend = INV_SRC_ALPHA.