High Dynamic Range Environment Mapping On Mainstream Graphics Hardware
by Lake, Northrop, Freeman

Contents   Introduction   Background   Theory   Implementation   Future Work   Appendix A   Source code   Printable version   Discuss this article

Theory

The Need for HDR Images

An image is composed of the response of each pixel of the sampling device. The light sensor has the largest error when near its maximum or minimum input. Since any value above the saturation point is mapped to the largest value for storage we have not obtained an accurate measurement of the amount of light hitting the pixel for a given exposure time. Therefore, in many cases images fail to accurately sample and store the intensities when HDR image techniques are not utilized. To compensate for the capabilities of today's digital cameras, it is useful to vary the exposure time and take a series of images. These images can then be used to gain a more accurate understanding of the light entering the lens and being sampled by the light sensor for later display as HDR images.

By storing a set of images with different exposure times it becomes easier to map the true luminance of a scene into the displayable range of whatever device or application limitations we have available. Much of the work in the film industry is motivated by the desire to have images that can be matched to the displayable range of film.

Uses of HDR images

HDR images are used for environment mapping of objects in a game engine. The motivation is to create a more accurate and compelling visual experience given the capabilities of today's hardware. In addition to HDR images being useful for environment mapping, they can also be used for motion blur and simulating characteristics of the human visual system when rendering. For example, it can be used in where the high luminance values are not clamped and used to demonstrate a depth of field effect [11], [8], [9].

Figure 3-1. HDR Rendering Sample
St. Peter's Light Probe Image © 1999 Paul Debevec. Used with Permission.

Figure 3-1 taken from our High Dynamic Range Environment Map demo. We use DirectX* 9.0 Pixel Shader Version 2.0 on Mobile Intel 965 Express Chipset graphics with 16-bit floating point textures and an RGBE image to store and render the HDR environment maps.

Display

To this point we've covered the acquisition, usage, and storage of HDR images. Next, it is important to consider the display of HDR imagery. There has been research in creating displays that can more accurately render HDR imagery [19]. Use of the currently available graphics technology allows floating-point formats Therefore, it is necessary to find techniques that allow mapping the large variance in luminance stored in HDR textures into something that can be displayed.

Given the presence of floating-point texture support in current hardware, luminance values are preserved without the High Dynamic Range Texture Mapping, HDRTM technique discussed in [2]. This eliminates the need to decode and encode 8-bit color values utilizing the Alpha channel as a temporary store for the RGBE luminance exponent and allows us to use the 16-bit per RGB and Alpha channel format leaving the Alpha channel unused.

Due to the real-time requirements and the wide availability of programmable graphics hardware, the authors have chosen to implement a tone-mapping technique presented in [17] for the demo seen in Figure 3-1. For tone mapping, the idea is to map the HDR of the real world luminance to the lower dynamic range of the display device. In fact, since pixels are always constrained to some maximum value in the frame buffer, we always have performed tone mapping by applying a clamp operator per pixel that causes our display to act as a low pass filter, thus we lost much of the higher luminance of the scene. We would like to take a smarter approach. Ansel Adams faced a similar problem in photography. We choose to adopt a technique inspired by his work called the Zone System. The Zone System is still widely used in analog image acquisition today.

As seen in Figure 3-2, a zone is a range of luminance values, taking into account the reflectance of the print. There are eleven print zones ranging from pure black to pure white, each doubling in intensity. Each zone is represented with a roman numeral, Zone 0* through Zone X. The middle-grey value is the subjective middle brightness region of the scene, which is mapped to print Zone V in most cases. A photographer would take luminance readings of what was middle grey in a scene, typically this middle grey would be what would produce an 18% reflectance on the print. If the scene were low key this value would be lower in the spectrum of print zones. Similarly, if it were high key the middle grey would be one of the higher print zones.

Figure 3-2. Zone System Luminance Intensity Scale

Figure 3-2 depicts the 11 Zones that comprise the Zone System of Ansel Adams as the intensity value varies from 0 to 255 increasing exponentially each step. Zone IX and Zone X both are saturated: We can only create images up to the value 255 and the Zone System contains values above what we can represent with a traditional image of 8 bits per pixel, therefore Zone X is clamped to 255.

Note: '0' was not a Roman numeral. In fact, the Romans had not yet discovered 0.

Mathematics of Reinhard's Photographic Tone Reproduction Operator

The first step in applying Reinhard's tone reproduction operator is to obtain an average luminance value, which we will use as the key of the scene. Normally, a simple average is fine, however since luminance values are an exponential curve we will use the logarithmic average luminance value, or log-average luminance as an approximation for the scene key. To compute this value, we first compute the luminance from our RGB values as follows:

Equation 1

Equation 1 uses the luminance conversion found in [1], which is based on modern CRT and HDTV phosphors.

Next, we compute the log average luminance using this value, summing over the entire image:

Equation 2

Here, the number of pixels is the total number of pixels in the image. The is a small value to avoid taking the log of a completely black pixel whose luminance is zero. Now that we have the image key we would like to re-map the pixels into a new image that scales the values so we can give greater dynamic resolution to the upper range of luminance values. Since we know that .18 is the middle (zone V) in our logarithmic scale from 0-1 in intensity values, we use the ratio:

Equation 3

Solving for the new luminance value we obtain:

Equation 4

Now, this is assuming our goal is to map the range such that zone V is in the middle. For images that have a higher than average image_key it may be desirable to raise the middle zone. To do this we generalize our expression:

Equation 5

Typically the midzone_luminance_value will vary in amounts that double each step: .045, .09, .18, .36, .72.

We still have two problems. First, most images have a normal dynamic range for most of the image and only a small percentage of the pixels have very HDR, for example at the light source such as the sun or a window. Equation 5 assumes a linear mapping, what we really want to do is a non-linear mapping to emphasize these areas of HDR. Second, Equation 5 can still produce values that lie outside of the 0.0 – 1.0 viewable on the monitor. This leads us to a final adjustment to our scaled luminance value:

Equation 6

Notice Equation 6 scaled our luminance values between 0 and 1, scaling high luminance values by very small amounts and doing nothing to low luminance values since the 1 dominates the calculation. This gives us a greater dynamic range in the high regions of luminance as can be see in Figure 3-3 and Figure 3-4.

Figure 3-3. Tone Mapped Luminance Scale

The image from Figure 3-2 modified by tone mapping. Notice now that all tones from 0 to X are in visible in the image and within the displayable range of 255 intensity values.

Figure 3-4. Tone Mapping Effects on Luminance

Luminance values before and after tone mapping algorithms in Equations 1-6 are applied. Code for this example is in Appendix A. For image_key we chose the value of 0.36.

For current real-time applications expressions 2, 5, and 6 are adequate and will bring all luminance values within a displayable range. Sometimes this is not always desirable and we would instead like them to burn out for certain values in the highest range of luminance. This is accomplished with a different tone mapping operator:

Equation 7

White_luminance refers to the smallest luminance value that will be mapped to pure white. Notice if white_luminance is large the fraction in the numerator of Equation 7 goes to 0 and we are left with Equation 6. However, if white_luminance is small we get larger values in the numerator of our expression, acting to enhance lower dynamic range pixels. In our examples, we use Equation 6.

Mapping Luminance to RGB

Next, to get the final RGB values we multiply the final luminance by each original RGB value respectively to compute the new pixel RGB values.

Conversion

The process to acquire a HDR image is relatively simple given the tools available today. A HDR image is constructed by collecting a set of conventional photographs with identical position and orientation and varying the exposure times, usually by varying the f-stop for each image. One way to create a HDR image is to implement the algorithm presented in [3] that can be used to recover the response function of the camera and use this information to construct a HDR image. This image will have pixels whose values represent the true radiance values in the scene. Another option is to use HDRShop, a tool that allows the manipulation of a set of low resolution images taken from a standard camera to be used to create a single HDR image [7].

For many applications of HDR images in entertainment where the incoming radiance at a point is important, a light probe is a more suitable format. A light probe is created by placing a mirrored ball in the environment and taking pictures from both sides. The result is an environment map: a set of ray samples (an image) of all the rays of light intersecting the point at the center of the light probe from each point in space. This can then be rendered into a high resolution sphere map or cube map for use in the rendering pipeline and is used to approximate the light rays intersecting the object we are environment mapping.

Implementation