Greetings,

I like matcap technique for it's simplicity. Before I used just view space normal as lookup but recently I found this article which uses a reflection vector and a fancy formula for lookup coordinates. What is the math behind this formula?

That's the spheremap transform (or, the 3D direction to 2D texture coordinate half of it). It even used to be part of the fixed function pipeline, back before cubemaps were standard. See: ftp://ftp.sgi.com/opengl/contrib/blythe/advanced99/notes/node177.html

You can use it with sphere normals, reflection directions, or any direction. It just maps the entire surface of a 3D sphere into a single 2D circle -- much like how a cubemap maps the entire surface of a 3D sphere into six 2D squares.

The right way to look up a matcap depends on how it's generated. Spheremapping allows the full sphere to be saved into the matcap, including the back. This is important for reflection maps, as the reflection vector for the edges of an object will point directly away from the camera.

Other times, maps might be authored for lookup using the view-space normal, e.g. for diffuse matcaps.

Yes, some might also be authored for lookup with a simple "uv = normal.xy*0.5+0.5" instead of a spheremap -- this is a hemi-spheremap: a transform between a half sphere and a 2D circle. There's also parabola maps, which do the same thing with different properties. These are most popularly seen used as a dual-parabola map, which encodes a full sphere instead of a hemisphere - e.g. http://graphicsrunner.blogspot.com.au/2008/07/dual-paraboloid-reflections.html

Thank you guys. But still not very clear. What is this formula does:

ftp://ftp.sgi.com/opengl/contrib/blythe/advanced99/notes/img166.gif

It looks like vector length, but what is this -1 term?

Yeah it's a kind of length formula... the +1 makes one side of the sphere lie on z=0, and the other side lie on z=2. That link is just an intermediate term in the whole formula. The full formula maps the front of the sphere into a circle, and the back of the sphere into a ring surrounding that circle. So the front of the sphere becomes the center of the image, the horizon becomes a circle about two thirds between the center and the edge, and the back of the sphere becomes a circle at the outer edges of the image.