Volumetric video

Volumetric video is a technique for capturing a real person, object, or scene as a dynamic three-dimensional representation that a viewer can look at from any angle. Unlike conventional flat video, which records a scene from one fixed viewpoint, a volumetric recording stores the 3D geometry and surface appearance of the subject over time, so it can be placed in a virtual or augmented environment and observed from a freely chosen position.

Because the output is a moving 3D model rather than a 2D image plane, volumetric video supports viewing with six degrees of freedom (6DoF), meaning the viewer can change both orientation and position. This distinguishes it from 360-degree video, where the viewer can only rotate the head (yaw, pitch, and sometimes roll) and the perspective stays fixed. Volumetric content is used as a source for virtual reality, augmented reality, and mixed reality experiences, as well as for visual effects, live broadcast, and on conventional 2D screens. The recordings are often described as "holograms" in marketing, though they are not holograms in the optical-physics sense.

History

The academic foundation of volumetric video is the work on free-viewpoint and "virtualized" capture of dynamic events in the 1990s. In 1997 Takeo Kanade and colleagues at Carnegie Mellon University described Virtualized Reality, a system that recorded a moving scene with a dome of 51 cameras (each 512 by 512 pixels, running at 30 frames per second) arranged around a five-meter geodesic dome. The system computed the 3D structure of the event using multi-baseline stereo and then rendered the scene from arbitrary new viewpoints by triangulation and texture mapping. This class of capture is referred to in the research literature as free-viewpoint video (FVV).

A widely cited step toward production-quality, streamable output came in 2015, when Alvaro Collet and colleagues at Microsoft Research published "High-Quality Streamable Free-Viewpoint Video" at SIGGRAPH (ACM Transactions on Graphics, volume 34, issue 4). The paper presented what the authors called the first end-to-end system to record performances with a dense set of RGB and infrared cameras, reconstruct dynamic textured surfaces, and compress them into a streamable 3D video format. Their capture rig used 106 calibrated and synchronized cameras, and the system encoded the result as tracked textured triangle meshes inside an MPEG stream so it could be played back in real time on consumer devices. This work became the technical basis for Microsoft's Mixed Reality Capture Studios.

Commercial capture facilities followed. In June 2018 the joint venture Volucap GmbH opened the first volumetric video studio on the European continent in Potsdam-Babelsberg, founded by ARRI, the Fraunhofer-Gesellschaft, Interlake System, Studio Babelsberg, and UFA. The studio is built around 32 cameras mounted on a light rotunda nearly four meters high and uses Fraunhofer Heinrich Hertz Institute software called 3D Human Body Reconstruction, which turns the captured footage into dynamic 3D models that can be processed like computer-generated ones.

How it works

A volumetric capture stage surrounds the subject with an array of synchronized cameras, typically several dozen to over a hundred, recording the subject from all sides at the same time. The cameras must be precisely calibrated so their images can be combined into a single consistent 3D coordinate space. Many systems pair colour (RGB) cameras with depth sensing, using infrared structured light, time-of-flight sensors, or laser-based depth sensing to measure the distance to surfaces. Reconstruction software then merges the colour, depth, and silhouette information to recover the shape of the subject for each frame and to apply the recorded colour as a texture.

Microsoft's stage is a representative example: the studio in San Francisco used 106 cameras, split into 53 RGB cameras that record colour video from many angles and 53 infrared cameras that read a pattern of laser dots projected onto the subject to map its surface. The combined feeds produced roughly 10 gigabytes of data per second, and the system devoted extra processing to faces so that expressions came through clearly. The active capture space was about eight feet in diameter and ten feet tall. Microsoft later built more portable mobile stages that use 64 cameras instead of 106.

Data representations and compression

The reconstructed result can be stored in several forms. A point cloud represents the surface as a set of coloured 3D points; a voxel grid divides space into small 3D cells; and a textured mesh represents the surface as connected triangles with an image atlas applied to them. Each frame of a volumetric recording is far larger than a frame of ordinary video, so compression is a central problem. The Collet system encoded tracked meshes into an MPEG video stream. For point-cloud content, MPEG standardized Video-based Point Cloud Compression (V-PCC) as part of ISO/IEC 23090-5, which converts each 3D frame into 2D image patches (called atlas data) so they can be compressed with existing video codecs such as HEVC. V-PCC is built on the more general Visual Volumetric Video-based Coding (V3C) framework.

Relationship to related techniques

Volumetric video is related to several other 3D-capture and rendering methods but is distinguished by its focus on moving, time-varying subjects.

Photogrammetry reconstructs a 3D model from a set of overlapping still photographs of a static object or scene. It produces a single fixed model rather than a moving one, so a volumetric capture can be thought of, in part, as photogrammetry-like reconstruction repeated for every frame of a performance.

Neural radiance fields (NeRF) use a neural network to learn a scene from images and synthesize photorealistic new viewpoints. NeRF was developed mainly for static scenes, and extending it to moving subjects (dynamic NeRF) is an active research area aimed at producing neural volumetric video with smaller data sizes than explicit meshes or point clouds. Gaussian splatting, a newer representation that depicts a scene as many overlapping 3D Gaussian blobs, is likewise being adapted from static capture toward dynamic, volumetric use.

Applications

VR, AR, and mixed reality

The most direct use of volumetric video in this field is placing a recorded person into an immersive scene. Microsoft Mixed Reality Capture Studios recorded performances that could be viewed in augmented reality, virtual reality, and on 2D screens; documented examples include the New York Times "Ashley Graham: Unfiltered" interactive AR piece and corporate keynote holograms. The company 8i built its business around streaming captured human "holograms" to phones, browsers, and VR or AR headsets, and in 2017 released a consumer AR app called Holo that let users place pre-recorded volumetric figures into their own photos and videos.

Sports and live events

Intel's True View system (originally developed by the Israeli company Replay Technologies as freeD) installs an array of high-resolution cameras around a stadium to produce volumetric replays. The system uses dozens of cameras based on 20-megapixel industrial sensors to build a 3D voxel model of the play, which a rendering engine can then show from any angle within the cameras' coverage; a single 15-to-30-second clip can involve up to a terabyte of source data. True View has been deployed in the home venues of around twenty National Football League teams and in soccer stadiums in Europe.

Current status

As of 2026 volumetric video is produced commercially by a network of specialist capture studios rather than by a single dominant platform. In August 2023 Microsoft announced that the volumetric specialist Arcturus had become the go-to-market partner for its Mixed Reality Capture Studios technology; the arrangement pairs creators with licensed MRCS studios (whose license holders include Metastage in Los Angeles, Dimension in London, and Jump in Korea) and combines the capture pipeline with Arcturus's HoloSuite editing and streaming tools. Volucap continues to operate its studio near Berlin, and 8i remains an active volumetric video company. Research effort has shifted toward neural and learned representations (dynamic NeRF and Gaussian splatting) and toward standardized compression such as MPEG V-PCC, with the shared goal of reducing the large data sizes that have limited streaming of volumetric content to consumer devices.