III. Theory

3.1 The need for HDR images

An image is composed of the response of each pixel of the sampling device. The light sensor has the largest error when near its maximum or minimum input. Since any value above the saturation point is mapped to the largest value for storage we have not obtained an accurate measurement of the amount of light hitting the pixel for a given exposure time. Therefore, in many cases images fail to accurately sample and store the intensities when HDR image techniques are not utilized. To compensate for the capabilities of today's digital cameras, it is useful to vary the exposure time and take a series of images. These images can then be used to gain a more accurate understanding of the light entering the lens and being sampled by the light sensor for later display as HDR images.

By storing a set of images with different exposure times it becomes easier to map the true luminance of a scene into the displayable range of whatever device or application limitations we have available. Much of the work in the film industry is motivated by the desire to have images that can be matched to the displayable range of film.

3.2 Uses of HDR images

HDR images are used for environment mapping of objects in a game engine. The motivation is to create a more accurate and compelling visual experience given the capabilities of today's hardware. Since integer texture formats are widely supported we demonstrate our HDR techniques using this format. This also demonstrates that floating point texture support is not a requirement for supporting high dynamic range imagery. This may provide just the right effect in key elements of the user experience. Since integer formats have a limited dynamic range, we chose a tone mapping operator that we will discuss for display on hardware that does not support floating point textures. In addition to HDR images as useful for environment mapping, they can also be used for motion blur and simulating characteristics of the human visual system when rendering. For example, it can be used in where the high luminance values are not clamped and used to demonstrate a depth of field effect [Northrop04], [Kawase03], [Kawase04].

Figure 4. Image taken from our High Dynamic Range Environment Map demo. We use DirectX 9.0 Pixel Shader Version 2.0 on Intel 915g Graphics and RGBE images to store and render the HDR environment maps. St. Peter's Light Probe Image © 1999 Paul Debevec. Used with Permission.

3.3 Display

To this point we've covered the acquisition, usage, and storage of HDR images. Next, it is important to consider the display of HDR imagery. There has been research in creating displays that can more accurately render high dynamic range imagery [Seetzen04]. However, this technology is in its infancy and not likely to be widely deployed in the next few years. Therefore, it is necessary to find techniques that allow mapping the large variance in luminance stored in HDR textures into something that can be displayed.

One such technique is called High Dynamic Range Texture Mapping, HDRTM, discussed in [Cohen01]. The basic idea is the following: since it is not possible to store the full range of luminance values in 8 bit per channel texture maps, instead break high dynamic range textures into a set of the 8 high bits and 8 low bits per channel. This technique is completely general: break the image down before sending to the hardware into corresponding channels, then reassemble and composite on the other side by inverting the decomposition process that was applied before submission to the graphics subsystem. This can be applied dynamically or statically depending on the goals of your application and assumes the presence of 16-bit texture values. Fixed point hardware techniques that use texture combining hardware and two textures simultaneously also exist.

Due to the real-time requirements and the wide availability of programmable graphics hardware, the authors have chosen to implement a tone-mapping technique presented in [Reinhard01] for the demo seen in Figure 4. For tone mapping, the idea is to map the high dynamic range of the real world luminance to the lower dynamic range of the display device. In fact, since pixels are always constrained to some maximum value in the framebuffer, we always have performed tone mapping by applying a clamp operator per pixel that causes our display to act as a low pass filter, thus we lost much of the higher luminance of the scene. We would like to take a smarter approach. Ansel Adams faced a similar problem in photography. We choose to adopt a technique inspired by his work called the Zone System. The Zone System is still widely used in analog image acquisition today.

As seen in Figure 5, a zone is a range of luminance values, taking into account the reflectance of the print. There are eleven print zones ranging from pure black to pure white, each doubling in intensity. Each zone is represented with a roman numeral, Zone 0* through Zone X. The middle-grey value is the subjective middle brightness region of the scene which is mapped to print Zone V in most cases. A photographer would take luminance readings of what was middle grey in a scene, typically this middle grey would be what would produce an 18% reflectance on the print. If the scene was low key this value would be lower in the spectrum of print zones. Similarly, if it was high key the middle grey would be one of the higher print zones.

Figure 5: The image above depicts the 11 Zones that comprise the Zone System of Ansel Adams as the intensity value varies from 0 to 255 increasing exponentially each step. Zone IX and Zone X both are saturated: We can only create images up to the value 255 and the Zone System contains values above what we can represent with a traditional image of 8 bits per pixel, therefore Zone X is clamped to 255.

* - '0' was not a Roman numeral. In fact, the Romans had not yet discovered 0.

3.4 Mathematics of Reinhard's Photographic Tone Reproduction Operator

The first step in applying Reinhard's tone reproduction operator is to obtain an average luminance value which we will use as the key of the scene. Normally, a simple average is fine, however since luminance values are an exponential curve we will use the logarithmic average luminance value, or log-average luminance as an approximation for the scene key. To compute this value, we first compute the luminance from our RGB values as follows:

    (1)

Equation 1 uses the luminance conversion found in [Akenine-Moller02] which is based on modern CRT and HDTV phosphors.

Next, we compute the log average luminance using this value, summing over the entire image:

    (2)

Here, the number of pixels is the total number of pixels in the image. The is a small value to avoid taking the log of a completely black pixel whose luminance is zero. Now that we have the image key we would like to re-map the pixels into a new image that scales the values so we can give greater dynamic resolution to the upper range of luminance values. Since we know that .18 is the middle (zone V) in our logarithmic scale from 0-1 in intensity values, we use the ratio:

    (3)

Solving for the new luminance value we obtain:

    (4)

Now, this is assuming our goal is to map the range such that zone V is in the middle. For images that have a higher than average image_key it may be desirable to raise the middle zone. To do this we generalize our expression:

    (5)

Typically the midzone_luminance_value will vary in amounts that double each step: .045, .09, .18, .36, .72.

We still have two problems. First, most images have a normal dynamic range for most of the image and only a small percentage of the pixels have very high dynamic range, for example at the light source such as the sun or a window. Equation 5 assumes a linear mapping, what we really want to do is a non-linear mapping to emphasize these areas of high dynamic range. Second, equation 5 can still produce values that lie outside of the 0.0 - 1.0 viewable on the monitor. This leads us to a final adjustment to our scaled luminance value:

    (6)

Notice equation 6 scaled our luminance values between 0 and 1, scaling high luminance values by very small amounts and doing nothing to low luminance values since the 1 dominates the calculation. This gives us a greater dynamic range in the high regions of luminance as can be see in Figure 6 and 7.

Figure 6. The image from figure 5 modified by tone mapping. Notice now that all tones from 0 to X are in visible in the image and within the displayable range of 255 intensity values.

Figure 7. Luminance values before and after tone mapping algorithms in equations 1-6 are applied. Code for this example is in Appendix A. For image_key we chose the value of 0.36.

For current real-time applications expressions 2, 5, and 6 are adequate and will bring all luminance values within a displayable range. Sometimes this is not always desirable and we would instead like them to burn out for certain values in the highest range of luminance. This is accomplished with a different tone mapping operator:

    (7)

White_luminance refers to the smallest luminance value that will be mapped to pure white. Notice if white_luminance is large the fraction in the numerator of equation 7 goes to 0 and we are left with equation 6. However, if white_luminance is small we get larger values in the numerator of our expression, acting to enhance lower dynamic range pixels. In our examples, we use equation 6.

Mapping luminance to RGB

Next, to get the final RGB values we multiply the final luminance by each original RGB value respectively to compute the new pixel RGB values.

3.5 Conversion

The process to acquire a high dynamic range image is relatively simple given the tools available today. A high dynamic range image is constructed by collecting a set of conventional photographs with identical position and orientation and varying the exposure times, usually by varying the f-stop for each image. One way to create a high dynamic range image is to implement the algorithm presented in [Debevec97] that can be used to recover the response function of the camera and use this information to construct a high dynamic range image. This image will have pixels whose values represent the true radiance values in the scene. Another option is to use HDRShop, a tool that allows the manipulation of a set of low resolution images taken from a standard camera to be used to create a single HDR image [HDRShop04].

For many applications of HDR images in entertainment where the incoming radiance at a point is important, a light probe is a more suitable format. A light probe is created by placing a mirrored ball in the environment and taking pictures from both sides. The result is an environment map: a set of ray samples (an image) of all the rays of light intersecting the point at the center of the light probe from each point in space. This can then be rendered into a high resolution sphere map or cube map for use in the rendering pipeline and is used to approximate the light rays intersecting the object we are environment mapping.

Implementation