Hi. My apologies if this discussion has been played out already. This topic seems to come up a lot, but I did a quick search and did not quite find the information I was looking for. I'm interested in knowing what is considered the best practice these days, with respect to deferred rendering and anti-aliasing. These are the options, as I understand them:

Use some post-processed blur like FXAA.

I've tried enabling NVidia's built-in FXAA support, but the results were not nearly acceptable. Maybe there is another technique that can do a better job?

Use a multi-sampled MRT, and then handle your own MSAA resolve.

I've never done this before, and I'm anxious to try it for the sake of learning how, but it is difficult for me to understand how this is much better than super-sampling. If I understand MSAA correctly, the memory requirements are the same as for super-sampling. The only difference is that your shader is called fewer times. However, with deferred shading, this really only seems to help save a few material shader fragments, which don't seem very expensive in the first place. Unless I'm missing something, you still have to do your lighting calculations once per sample, even if all of the samples have the same exact data in them. Are the material shader savings (meager, I'm guessing) really worth all of the hassle?

Use Deferred Lighting instead of Deferred Shading.

You'll still have aliased lighting, though, and it comes at the expense of an extra pass (albeit depth-only, if I understand the technique correctly). Is anybody taking this option these days?

Use TXAA

NVidia is touting some TXAA technique on their website, although details seem slim. It seems to combine 2X MSAA with some sort of post-process technique. Judging from their videos, the results look quite acceptable, unlike FXAA. I'm guessing that the 2X MSAA would be handled using your own custom MSAA resolve, as described above, but I don't know what processing happens after that.

These all seem like valid options to try, although none of them seem to be from the proverbial Book. It seems to me, though, that forward rendering is a thing of the past, and I would love to be able to fill my scene with lights. I could try implementing all of these techniques as an experiment, but since they each come with a major time investment and learning curve, I was hoping that someone could help point a lost soul in the right direction.

Bonus questions: Is there a generally agreed-upon way to lay out your G-Buffer? I'd like to use this in conjunction with HDR, and so memory/bandwidth could start to become a problem, I would imagine. Is it still best practice to try to reconstruct position from depth? Are half-float textures typically used? Are any of the material properties packed or compressed in a particular way? Are normals stored as x/y coordinates, with the z-coord calculated in the shader?

I'm using OpenGL, if it matters.

I use FXAA but I don't use whats built in with the graphics driver. What you can do for better results is download the FXAA 3.9 shader (used to be on Timothy Lottes blog but I can't find it anymore), it has some conditional compilation setup in it which you can use to tweak the settings. This method is far better then using the graphics driver because you can apply it at a more optimal spot in your rendering pipeline (preventing some unwanted blur). Amazingly the same shader works for both hlsl and glsl and it will work on Intel and AMD gpus as well as consoles. It is important to note that you must have some method to generate luminosity before running the fxaa shader (this is pretty trivial).

Thanks so much for your replies.

If I download the FXAA 3.9 shader and integrate it in my pipeline, will it help beyond allowing me to avoid blurring HUD/text elements? The reason I ask is that I have a lot of objects in my scene with long, straight edges -- particularly buildings and chain link fences, but also some vehicles as well -- with which FXAA seems to work particularly poorly. At a distance, these objects create wild, shimmering jaggies that are very distracting. Will downloading the shader actually improve this? Here are a couple examples:

As you move the viewpoint around, those broken lines crawl and shimmer.

I've spent the last couple hours reading through the links that you provided, MJP, and it has given me a lot of food for thought.

I'm particularly intrigued by the Forward+ idea, because the idea of using an MRT with HDR and MSAA is starting to sound prohibitive. Let's say I use the G-Buffer layout that you mentioned in your March 2012 blog post on Light-Indexed Deferred rendering, except the albedo buffers need to be bumped up to 64bpp to accommodate HDR rendering (right?). Then, multiply the whole thing by 4 for 4x MSAA, and I have a seriously fat buffer. And what do I do about reflection textures? If I want to do planar reflections or refractions, for example. That seems like it'd be another big fat g-buffer. Am I thinking about this correctly? Plus, you have the lack of flexibility with material parameters that comes with deferred rendering.

Edit: On the other hand, isn't it somewhat expensive the loop through a runtime-determined number of lights inside a fragment shader? If it isn't, then why did the old forward-renderers bother compiling different shaders for different numbers of lights? Why did they not, instead, just allow 8 lights per shader (say), and use a uniform (numLights) to determine how many to actually loop through? Sure, you only get per-object light lists that way, which is imprecise, but is it really slower than having a separate compute shader step that determines the light list on a per-pixel basis?

But if I do end up going the Deferred w/ MSAA route (which I'm kind of leaning toward), using edge detection to find the pixels I actually want to do per-sample lighting on, and doing per-pixel lighting for all others, sounds like it will be a huge time-saver, even if I have to eat a huge amount of memory.

And in any case, the information you provided helped me discover all sorts of inefficiencies with the way we're doing deferred shading, so it looks like I can probably gain some speed just with a few optimizations.

Also, if you don't actually integrate it into your pipeline, then it's not actually in your game. It's pretty unkind to your users to say "to make my game look best, go into your nVidia driver panel (sorry ATI/intel users) and enable these hacks".

Hah, good point. This isn't a product that I plan on releasing into the wild, so that's not even something that I considered.

On more modern cards, branching is less costly. The biggest worry is when nearby pixels take different branches (e.g. one pixel wants to loop through 8 lights, but it's neighbour wants to loop through 32 lights) -- but most of these Forward+-ish techniques mitigate this issue by clustering/tiling pixels together, so that neighbouring pixels are likely to use the same loop counters and data.

Could I trouble you for more information about this? Why is it that nearby pixels like to have the same number of pixels, and what is meant by "nearby"? Would an 8x8 tile suffice? (that seems to be the common size, from what I've been reading)

The power cables in our scene are basically just rectangular strips, onto which a texture of a catenary-shaped wire is mapped. This was done by a modeler long before my time, so I don't know why he decided to do it that way, but one fortunate side-effect is that the texture filtering takes care of everything for me. The fence poles and light posts are a different story, but because they never end up being much smaller than a pixel, the 4x MSAA seems to do an okay job on them. Thanks for the PDF, though, it has a lot of useful information.

Hodgman, thanks again for your help. I'm much obliged to everyone here. Although I must say, you guys are only encouraging me to ask more questions. =)