Outline shaders are always difficult to pull off. It is hard to get fine outlines done (most edge detection algorithm like sobel result in a atleast two pixel thick line). In combination with a high contrast color like black, the outline effect will be just too harsh and therefor many people will just dislike the effect, in games atleast.
I play around with a simple sobel filter to get the edge detected done. As detection criteria I use depth and normals. Currently I'm not using any AA filters, so it will look really clumpsy. In my first screenshot (upper part) you see a standard sobel edge detection in pure black. As you can see, it has an other negative effect, any bloom effect will be interupted. To counter this effect I fade out the outlines with distance (lower part, right arrow) and use a blurred,darkened version of the scene as input of the outline pixel. This has the advantage that bloom works (gets a bright outline) and the lines has lot less contrast compared to pure black.
An other challenge with outline shaders are particles. My next screenshot(upper part) shows clearly outlines behind the fire particle effect. This is really annoying, to get around it, I render the particle effects on a separate render target (btw. half resolution) and use the alpha channel of this render target as alpha mask for the outlines. You can compare it with the bottom part of the screenshot where you can see the effect of masking the outlines. The alpha mask is displayed in the bottom,right corner.
Next I will take a look at AA filter (FXAA, MLAA, ...). Here are some comparison shots displaying the game without (upper part) and with (lower part) outline effect applied.