I'm just about finishing funtionally with my game, and I'm looking to add one or two nice touches. One thing I would like to be able to do is draw a colored outline around certain objects. Now the important thing to know here is that I might be drawing multiple objects with outlines at any given time and the color need not always be the same. I'd like to outline to be a consistent thickness. Because the outline should be of consistent thickness, I'm thinking that doing this in a mesh shader is probably a bad idea. I could probably knock something up using the view vector and the surface normal and getting a rough outline that way - kinda like low grade cel shading outlines used to be done - but it's going to be dependent on the view, and so I'd have to adjust by the distance between the camera and the mesh, and it probably won't look good at certain depths. So I'm thinking this is probably best done as a post process shader. Then it's in screen space and the thickness of the outline will be constant. The only time it will look bad is when the object is a very, very long way away, and the game doesn't allow this to happen in the first place. Now performance in my game is good, but I don't have a lot to spare, so I need this to be as quick as possible. I don't need particularly high quality and I don't need an outline on sharp internal edges or anything like that so I don't think I need to render out normals and depths to a rendersurface. I would imagine a very simple shader to just output a flat color would be fine for what I'm going to call the "stencil" pass, where I just draw everything which needs an outline. But I would then need to apply a shader which calculates the outline. I've seen a couple of HLSL shaders to do a sobel filter, which I think results in an image which is black and white. That's fine for a single color outline, but if I want a blue outline on one mesh and a red outline on another mesh, that won't work. I considered perhaps rendering twice, once to a "stencil" surface which simply indicates where the outline meshes are, and another which outputs the color of the outline. Then I could filter the black and white one, sample both, and multiply them to get the output color. But the sobel filter is necessarily going to highlight pixels which were slightly outside the original mesh and therefore are not colored on the color map, so that won't work. Perhaps there's a simple solution I'm missing, but I can't really come up with a fast, straightforward way of getting coloured outlines around my meshes. I'm using TrueVision3D, by the way, so I can use my own .fx shaders and I know a little bit of HLSL. Hopefully someone can spark an idea for me.

You can do a sobel filter on multiple color channels. So for your "stencil" texture, if you were to render the mesh that need outlines using the desired color you would then end up with that color outlines for your sobel pass.

Ooh, that's a pretty neat idea. Yeah, I'm only ever going to need three different outline colors, so that ought to work very nicely. Thanks Matt!

No problem, let me know if you have any trouble.

Also, if you need some sample shaders to look at you can check out the non-realistic rendering sample on the XNA CC site (I know you're not using XNA, but the shaders are all the same since it's HLSL).