Interesting! Where can I find more information on implementing something like that?

I think this blog post http://blog.selfshadow.com/2011/07/22/specular-showdown/ may also help you.

Holy shit! Enabling the Toksvig map in this test application makes it look extremely good during magnification! Something that's always bothered me was how smooth surfaces far away looked, but with a Toksvig map it almost looks exactly as I'd expect during minification! The shader there seems to be tuned for Blinn-Phong which uses a specular exponent instead of roughness. How do I this for Cook-Torrance and roughness?

EDIT: I NEED THIS. I've been staring at this for so long now.

It's very similar for Cook-Torrance. Check out great sample by MJP which implements various specular AA techniques with Cook-torrance: https://mjp.codeplex.com/releases/view/109905

Is this really the correct result?

It's correct assuming that there is no masking (self-shadowing on a sub-pixel scale) in that direction. Rough surfaces would block some of that light.

edit: late to the party, and also hit post to quick:

http://www.unrealengine.com/files/downloads/2013SiggraphPresentationsNotes.pdf

Has some good points on how to pre-integrate the BRDF for varying surfaces of roughness to mask the specular light.

I haven't been this excited about graphics in a long time! I've been thinking about this non-stop since yesterday. >___> I'm gonna spell out what I've managed to figure out so far.

So Toksvig is actually pretty simple. The basic idea is to analyze how much the normal changes from pixel to pixel in the normal map by blending together normals of neighboring pixels and checking the length of the resulting vector. If the resulting length is close to 1.0 the normal is constant across the tested area and no aliasing will likely occur, so we don't need to do anything. A vector with a length less than 1.0 indicates that the normal changes rapidly across the tested area, since averaging together two normals that point in different direction results in a vector with a length less than 1.0. This is for the same reason why we need to re-normalize our vertex normals in the fragment shader after they've been interpolated across a triangle.

When a rapid change in the normal map is found, Toksvig modifies the specular power / roughness of the fragment in an attempt to significantly increase the size of the specular highlight to prevent aliasing. The idea is that if we have a short normal, it originally consisted of many normals which pointed in different directions. In other words, due to the normal map it now has a significant increase in roughness due to the spread of the normals. I haven't taken a look at MJP's source code yet (I don't code in C/C++ so there's that as well), but exactly how to increase the roughness to achieve the intended effect is what I'm trying to figure out next.

On a related note, I believe I may have discovered a major problem with my normal maps. My normal maps are currently compressed using GL_COMPRESSED_RG_RGTC2 (AKA BC5 compression), which means that I only store X and Y of the normals and reconstruct Z in my fragment shader. The problem I've found is that I let OpenGL generate the mipmaps for me using glGenerateMipmap(). As described above, averaging together normals will make them shorter, but my fragment shader assumes a normal vector length of 1.0 to reconstruct Z! My normal maps are probably being significantly corrupted by this!

I will attempt to address both the specular aliasing problem and my normal map mipmap problem at the same time. The idea is as follows:

1. Load in the normal map from file into a 16-bit float RGB texture and normalize the imported 8-bit normal vectors for maximum precision.

2. Generate mipmaps as before by simply averaging together 2x2 pixels (no normalizing).

3. Generate a Toksvig mipmap chain by processing each mipmap level of the normal map with this shader.

4. Bake together the newly generated Toksvig maps with the roughness maps I already have.

5. Finally go through all mipmap levels in the normal map again and normalize all the normals in them, then compress to GL_COMPRESSED_RG_RGTC2.

The output is a compressed normal map with renormalized vectors to make it possible to accurately reconstruct Z in the fragment shader, and a new roughness map which will prevent specular aliasing.

The only problem I have now is figuring out exactly how to modify the roughness value I have from my roughness map to prevent aliasing. I'll go take a look at MJP's source code and see what I can figure out.

Keep in mind that normal maps aren't the only source of aliasing. Fine geometric detail will also be prone to aliasing once triangles become too small relative to pixels. I've had some success using custom MSAA resolves to combat this issue, but for a lot of people MSAA isn't an option. Temporal antialiasing can also help with or without MSAA.

I'm currently using a combination of 2x temporal supersampling with reprojection and motion vector weighting, FXAA and basic ordered grid supersampling to achieve pretty good anti-aliasing, and I have further plans to improve the anti-aliasing for geometry. I completely understand that Toksvig does not help at all when it comes to geometry aliasing.

Is Toksvig the best algorithm when you don't have worked on a custom algo ?

It's bad that nvidia DXT photoshop plugin doesn't generate the toksvig.

Is it not too heavy to have all normal map using R16G16 ?

Is Toksvig the best algorithm when you don't have worked on a custom algo ?

It's bad that nvidia DXT photoshop plugin doesn't generate the toksvig.

Is it not too heavy to have all normal map using R16G16 ?

Depending on your target platform it's fine, if you're targeting say new consoles/high end PC's then it's a good choice to get nice clean normals.

If you want flexibility you can always have a fallback path for lower quality if it's your thing.