Volumetric Capture and Photogrammetry

This article will explain in detail what is volumetric video and photogrammetry and how it is used for games, VR, AR. Photogrammetry has been around since the mid-19th century. The first iterations of the technique were mainly used in measuring and interpreting photographic images, creating topographic maps as well as patterns of the electromagnetic radiant imagery.

Over the years, the technology evolved into more sophisticated uses. Later variants of photogrammetry were applied in extracting 3D measurements from 2D data such as photos. It also has been used to extract other rich snippets of information from photos like the color ranges and values such as ambient occlusion, albedo or metallicity. This data is subsequently used for physical rendering.

Contemporary uses of the traditional photogrammetry now allow for the extraction of more granular information from images. It is starting to find relevance with emerging technologies such as virtual reality.

How it works?

Contemporary photogrammetry involves taking many images or using multiple cameras rig set in a spherical formation to capture the scene. Every camera in the rig is mounted at an angle so that its field of view can overlap the fields of view of the surrounding cameras. The overlap enables editors to generate seamless and continuous footage without gaps. There are professional photogrammetry cameras that can also be used for the same purpose and which work just like the hand-rigged setups.

After the filming, editors stitch up the footage to create a single and continuous experience. They do this by using a photogrammetric software like Reality Capture. It detects and matches points of interest in the images. When points match, it creates a point in 3D space which will also include color information. There are millions of such points in an image.

This sparse cloud represents the core of the photogrammetric process. It can be passed through other systems and be transformed into a dense point cloud or even a high poly mesh. It can also be optimised and then refined to get rid of the points which aren’t considered as noise.

The physical camera formation used in photogrammetry might seem like an archaic way of capturing immersive footage. Actually, this capture method is widely relied upon. However, photogrammetry is only able to capture still and not moving objects. It is great for large VR environments as without any modeling you can create walk through experiences. However, technology called volumetric video allows you to capture moving objects. It brings the most important detail into VR - real people.

Both photogrammetry and volumetric capture leverage images from sensors and cameras to create 3D point cloud. It can be seamlessly inserted into game engines, virtual reality experiences and augmented reality. However, photogrammetry is used for environments and volumetric video is for moving objects, characters.

The principle of photogrammetry.

The principle technique in photogrammetry is triangulation which entails forming lines of sight from two images. The lines of sight can subsequently be mathematically intersected to generate 3-dimensional coordinates of the reference points in the image.In photogrammetry, still images are processed to generate a high-resolution 3D mesh. The resulting 3D mesh is a complex framework that can be used to construct low-res game-ready environments.

The main limitation of photogrammetry as a content creation technique is that the outcome is static.The shots must also be edited post-production which might take a long time to do.

Why Volumetric Capture is a Game Changer

Volumetric capture technology also involves the use of multiple cameras to capture the footage of a person from different viewpoints. This footage is subsequently analysed by a piece of software. It stitches the frames in real time and recreates the viewpoints of the footage into a fully volumetric human.

Volumetric capture also entails creating a point cloud but with the motion factor slotted in. In volumetric video, the point cloud is generated from a sequence of frames. In this way, volumetric capture can record not only a location but also a performance or sequence of movements. It has a time factor unlike the static 8-bit graphics in photogrammetry. Thus, volumetric capture can be used in creating very realistic interactive characters.

Volumetric Capture Workflows

Volumetric videos can be generated using either point-based workflows or mesh-based workflows (like in photogrammetry). These workflows are not mutually exclusive; you don’t have to choose one or the other. You can use them in combination to generate volumetric videos.

Point-based workflows are gaining greater currency over the traditional 3D mesh when it comes to volumetric capture. In this workflow, the final data is packaged as points or particles in 3D space with point size and colour information included in them. The point-based workflow enables greater information density as well as content of very high spatial resolution. It has very high data rates which still can’t be handled adequately by the current graphics cards that have been more accustomed to the mesh-based pipeline.

Once the volumetric data has been captured it is edited and processed using software. Most of them are proprietary and not available to the public. However, currently one company EF EVE has a publicly available subscription. It can automatically calibrate and process data from up to 10 sensors. The final volumetric capture product can then be viewed via a virtual reality headset, AR or simply on the Web.

As the immersive world gradually becomes the new normal, volumetric capture will become the default technique for 3D content creation. The technology is evolving at a rapid rate and different volumetric capture companies are emerging. They have unique approaches that aim for simplicity and efficiency in the generation of volumetric video capture.