Light field capture
More actions
Light field capture is the photographic recording of a light field, so that new views of a real scene can be computed after the images have been taken. Light field is the set of light rays flowing through a point, surface or volume from every direction. Light field capture seizes all of the information about those rays of light. Using the information, you can generate an image that can be viewed from any direction. You can also generate a stereoscopic image, an image for each eye to create a 3D effect. For example, imagine an apple in the middle of a room. With light field capture, you can view the apple from any angle, top to bottom and front to back. With regular photography you can only view the apple from one direction.
In practice a light field is sampled rather than recorded completely: a camera is moved through many positions, many cameras are fired together, or an array of small lenses is placed in front of a single sensor. Marc Levoy of Stanford University described the resulting data as "a 2D collection of 2D images of a scene", and computing a new perspective view as extracting a 2D slice from that 4D array of pixels.[1] The spacing and resolution of those images set how far a viewer can move and how sharp the result looks.[1]
With these properties, light field capture has great possibilities in Virtual Reality. It can be used to create live action, VR environments that are rendered in real time. A captured light field reproduces motion parallax and view-dependent effects such as reflections as the viewer's head moves, which 360-degree video shot from a single point cannot do.[2][3] The viewer can move only within the volume that the cameras sampled.[4]
OTOY described its May 2015 demonstration as the first ever light field capture for Virtual Reality.[5][6] Lytro and Google later built their own light field camera systems for VR stills and video.[7][3]
Definition
Levoy's 2006 survey describes the underlying quantity as radiance, the amount of light traveling along a ray. The radiance along every ray in a region of 3D space lit by an unchanging arrangement of lights is called the plenoptic function; because a ray can be described by a position (x, y, z) and two angles, the plenoptic function has five dimensions.[1] Outside the convex hull of the objects in a scene, radiance stays constant along a ray, so one dimension is redundant. The remaining four-dimensional function is what Levoy and Pat Hanrahan call the 4D light field, defined as "radiance along rays in empty space".[1]
A common way to parameterize the 4D light field is by where each ray crosses two planes. Levoy notes that a two-plane light field can be thought of as a collection of perspective images of one plane, each taken from an observer position on the other plane.[1] This is the link between the mathematical light field and photography: an ordinary camera placed at many positions records exactly such a collection.
Levoy and Hanrahan's 1996 paper states that the light field "completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination", and that once it has been captured, "new views may be constructed in real time by extracting slices in appropriate directions", without depth information or feature matching.[8]
History
Early theory
According to Levoy, Michael Faraday proposed in an 1846 lecture that light should be interpreted as a field. Arun Gershun defined the light field concept in a 1936 paper, as the amount of light traveling in every direction through every point in space; working before digital computers, Gershun had no way to measure one.[1] The use of lens arrays to record light fields has its roots in Gabriel Lippmann's 1908 invention of integral photography, the ancestor of integral imaging.[1] In 1992, Adelson and Wang published "Single lens stereo with a plenoptic camera" in IEEE Transactions on Pattern Analysis and Machine Intelligence.[9]
Light fields in computer graphics
Light fields entered computer graphics in 1996 with two papers presented at SIGGRAPH 96: "Light Field Rendering" by Levoy and Hanrahan of Stanford University and "The Lumigraph" by Steven Gortler, Radek Grzeszczuk, Richard Szeliski and Michael Cohen of Microsoft Research.[8][10] The Lumigraph paper describes a method for "capturing the complete appearance of both synthetic and real world objects and scenes" and rendering them from new camera positions; its Lumigraph is "a subset of the complete plenoptic function".[10] Levoy later wrote that adding an approximate 3D model of the object to a light field greatly reduces the number of images needed, and that pure light field rendering and model-based rendering are two ends of a continuum indexed by how much geometry is known about the scene.[1]
The Stanford Computer Graphics Laboratory built several capture devices, including a spherical gantry with four motorized motions, a large camera array, a plenoptic camera and a light field microscope.[1] A light field of Michelangelo's Night in the Medici Chapel, captured in 1999, contained 24,000 images of 1.3 megapixels each, the largest Levoy knew of in 2006.[1] Bennett Wilburn and colleagues described a system of 100 custom video cameras in 2005, used among other things as a video camera with a large synthetic aperture.[11]
Hand-held plenoptic cameras
In a 2005 Stanford technical report, Ren Ng, Levoy and colleagues presented "a camera that samples the 4D light field on its sensor in a single photographic exposure", built by inserting a microlens array between the sensor and main lens. The images could be refocused after capture.[12] Lytro began as a maker of point-and-shoot cameras that could refocus after capture, and later turned its light field technology toward virtual reality.[13] A 2021 review in Visual Computing for Industry, Biomedicine, and Art names Lytro and Raytrix as companies that industrialized light field acquisition devices.[14]
Capture methods
Summarizing an earlier survey by Wu and colleagues, Zhou et al. group light field acquisition into three approaches: multisensor capture, which uses several cameras at once (most often a camera array); time-sequential capture, which uses one camera over multiple exposures and is time consuming; and multiplexed imaging, which encodes the higher-dimensional data into a single 2D image and is the most popular.[14]
| Approach | How it works | Trade-offs | Examples |
|---|---|---|---|
| Moving camera (time-sequential) | One camera is moved across a plane, cylinder or sphere and photographs the scene from each position[1] | Only works for static scenes; capture takes seconds to minutes[1][3] | Stanford spherical gantry; OTOY 2015 rig; Google 16-camera and two-camera rotating rigs |
| Camera array (multisensor) | Many cameras fire together, so a dynamic scene can be recorded[1] | Needs many synchronized cameras and large amounts of data[7] | Stanford multi-camera array; Lytro Immerge; Google 46-camera light field video rig |
| Lens array (multiplexed) | A microlens array in front of one sensor records many slightly different views in a single exposure[1][12] | Spatial resolution is traded for angular resolution; the viewpoints span only a short baseline[1] | Stanford hand-held plenoptic camera; Lytro consumer cameras; Raytrix cameras[14] |
Levoy illustrates the lens-array trade-off with Stanford's prototype plenoptic camera, which had a 16-megapixel sensor, an f/4 main lens and 300 x 300 microlenses: it could refocus anywhere within the depth of field of an f/56 camera, but the computed images were only 300 x 300 pixels.[1] For VR the bigger problem is the size of the viewing volume. Google researchers wrote in 2018 that lenslet arrays "typically capture too small a viewing volume for VR", noting that earlier microlens-array approaches allowed viewpoint changes of only a few millimeters, and that motorized gantries are "usually slow or optimized for planar or inward-pointing data".[3] The VR systems described below use rotating cameras or camera arrays spanning tens of centimeters.
Sampling density matters for all methods. Levoy notes that if the images have too few pixels, renderings become blurry as the viewer moves away from the original camera positions, and if there are too few images, the renderings "contain ghosts arising from blending different views of an object".[1] Google's 2018 system reduces such ghosting by rendering with per-view geometry derived from depth maps and by prefiltering the light field.[3]
Light field capture for VR
| System | Developer | First shown | Capture hardware (as described) | Content |
|---|---|---|---|---|
| Prototype light field rig | OTOY | May 2015 | One Canon DSLR with a Canon 8-15mm lens on a rig rotating around two axes; a second camera acted as a counterweight[15] | Still scenes |
| Lytro Immerge | Lytro | November 2015 | Announced as a sphere of modular rings holding hundreds of cameras, with a dedicated server[7] | Video |
| Lytro Immerge 2.0 | Lytro | November 2017 | 95 cameras in alternating rows pointed in opposite directions[16] | Video |
| 16x GoPro rig | March 2018 | Vertical arc of 16 GoPro Hero4 cameras on a rotating platform[3] | Still panoramas | |
| 2x DSLR rig | 2018 | Two Sony a6500 mirrorless cameras with 8mm Rokinon fisheye lenses on an 80 cm pivoting platform[3] | Still panoramas | |
| Light field video rig | 2020 | 46 time-synchronized action cameras on a hemispherical dome 92 cm in diameter[17] | Video at 30 fps |
OTOY (2015)
OTOY announced at the FMX 2015 conference in Stuttgart on 7 May 2015 what it called "the first ever capture of a spherical light field of a real world environment" used to produce a navigable scene in VR. Paul Debevec, OTOY's chief scientific consultant, presented the process in a session titled "Light Fields for Virtual Reality". The company said its ORBX media format compressed the data enough to keep file sizes manageable at the frame rates VR needs. OTOY chief executive Jules Urbach pointed to virtual tourism, such as stepping inside the Great Pyramid, as a use.[5]
UploadVR's explanation of the prototype said it used a single Canon DSLR with a Canon 8-15mm lens, rotated around two axes while taking stills over several minutes, so it could only record static subjects.[15] Engadget noted that the published video showed a viewer tilting their head from a fixed position rather than walking through the space.[18] In October 2015 Road to VR tried OTOY's captured light fields in an HTC Vive. Ben Lang wrote that the scene looked correct however he rotated or moved his head inside the captured volume, but that "if you move your head outside of the area of the box, the world goes blank". OTOY told the publication that its viewer ran at 2,000 FPS on PC and 120Hz on mobile, and that it could fit a 1-meter light field cube into 5-36 Mb.[4]
Lytro Immerge (2015-2018)
Lytro announced Immerge in November 2015 as a light field camera for cinematic VR that could record motion rather than stills. As announced, the system was a sphere of modular rings containing hundreds of cameras, with a server to handle data that UploadVR described as measured in terabytes per hour, editing tools that worked with Nuke, and a player for position-tracked playback on PlayStation VR, HTC Vive and Oculus Rift. Lytro claimed six degrees of freedom within about one meter and planned prototypes for the first quarter of 2016.[7] In August 2016 Lytro showed the first footage captured with Immerge, which it called the "first piece of 6DOF 360 live action VR content ever produced". Road to VR reported that, according to Lytro vice president of engineering Tim Milliron, Immerge synthesized each view from hundreds of constituent sub-cameras.[19]
Immerge 2.0, revealed in November 2017, had 95 cameras. Its alternating rows of cameras pointed off-axis in opposite directions widened each capture from 90 to 120 degrees, so a full 360-degree scene took three rotations instead of five.[16][20] Road to VR and DPReview reported that output resolution rose from 3.5K to 5K per eye, with Lytro saying the system was ready for 10K per eye once headsets could display it; calibration, previously manual, was automated.[20][16] On 27 March 2018 Lytro announced that it would stop taking new productions or providing professional services and would wind down the company. Google hired some Lytro staff but did not acquire the company.[13]
Google Welcome to Light Fields (2018)
On 14 March 2018 Paul Debevec, then a senior researcher at Google VR, described Google's light field experiments. The team rebuilt a GoPro Odyssey Jump rig into a vertical arc of 16 cameras on a rotating platform. In about a minute it recorded roughly 1,000 outward-facing viewpoints on a 70 cm sphere. Captured locations included the Gamble House in Pasadena, the Mosaic Tile House in Venice, St. Stephen's Church in Granada Hills and the flight deck of Space Shuttle Discovery at the Smithsonian. Google released the results as Welcome to Light Fields, a free app on Steam for HTC Vive, Oculus Rift and Windows Mixed Reality headsets.[2]
The accompanying SIGGRAPH Asia 2018 paper gives more detail. The GoPro rig spins in 30, 60 or 120 seconds and records 16 rows of 72 views; a second rig swings two Sony a6500 mirrorless cameras through a spiral and can shoot 7,776 bracketed high-dynamic-range photos in 33 minutes. The GoPro rig's 70 cm sphere of camera positions gives a viewing volume about 60 cm across, chosen to allow seated head movement. A modified VP9 video codec compresses each light field still to 50-200 MB while keeping random access, and the renderer produces stereo views at 90 Hz on consumer VR hardware. The authors reported more than 15,000 downloads of the app.[3]
Google light field video (2020)
At SIGGRAPH 2020, Michael Broxton, Debevec and other Google researchers presented a system for capturing and streaming light field video. It uses 46 time-synchronized action cameras on a 92 cm hemispherical dome and supports an 80 cm viewing baseline, more than 220 degrees of field of view, 10 pixels per degree and 30 frames per second, reconstructing objects as close as 20 cm to the rig.[17] The scene is reduced to a small number of RGBA and depth layers that are packed into texture atlases and compressed with standard video codecs, which Road to VR reported made the format light enough to render on mobile VR and AR platforms or in a web browser.[21][17]
Relation to other capture techniques
Overbeck and colleagues contrast panoramic light fields with omnidirectional stereo, which gives stereo views in all directions from a single viewpoint, and with concentric mosaics, which allow movement within a circle on a 2D plane. Their panoramic light fields give full six degrees of freedom (three of rotation, three of translation) inside a limited viewing volume.[3] Light field rendering does not require a 3D model of the scene, unlike methods such as photogrammetry that reconstruct one first; Levoy describes the two as ends of a continuum indexed by how much geometry is known.[1]
Machine learning has reduced how many views a capture needs. Local Light Field Fusion (2019), by Ben Mildenhall and colleagues, expands each input view into a local light field using multiplane images and blends nearby ones; the authors report the perceptual quality of Nyquist-rate sampling "while using up to 4000x fewer views", and built an augmented reality smartphone app that guides hand-held capture.[22] Neural Radiance Fields (2020) represent a scene with a neural network that takes a 5D coordinate (position and viewing direction) and outputs density and view-dependent radiance, trained from ordinary photos with known camera poses.[23] 3D Gaussian splatting (2023) represents the radiance field with 3D Gaussians and renders new views in real time, at 30 fps or more at 1080p, after training from multiple photos or videos.[24] These methods start from the same kind of multi-view photographs that light field capture records.
On the output side, the 2021 review names Avegant, Leia, Light Field Lab, Dimenco, CREAL and Looking Glass Factory in its discussion of light field display products.[14]
See also
References
- ↑ 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 Marc Levoy (August 2006). "Light Fields and Computational Imaging". Computer (IEEE Computer Society), vol. 39, no. 8. pp. 46-55. doi:10.1109/MC.2006.270. https://graphics.stanford.edu/papers/lfphoto/levoy-lfphoto-ieee06.pdf. Retrieved 2026-09-27.
- ↑ 2.0 2.1 Paul Debevec (2018-03-14). "Experimenting with Light Fields". The Keyword (Google blog). Google. https://blog.google/products-and-platforms/products/google-ar-vr/experimenting-light-fields/. Retrieved 2026-09-27.
- ↑ 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 Ryan S. Overbeck, Daniel Erickson, Daniel Evangelakos, Matt Pharr, Paul Debevec (November 2018). "A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality". ACM Transactions on Graphics, vol. 37, no. 6, article 197 (SIGGRAPH Asia 2018). doi:10.1145/3272127.3275031. https://arxiv.org/abs/1810.08860. Retrieved 2026-09-27.
- ↑ 4.0 4.1 Ben Lang (2015-10-20). "Exclusive: OTOY Shows Us Live-captured Light Fields in the HTC Vive". Road to VR. https://roadtovr.com/exclusive-otoy-shows-us-live-captured-light-fields-in-the-htc-vive/. Retrieved 2026-09-27.
- ↑ 5.0 5.1 "OTOY demonstrates first ever light field capture of a real world environment, producing photorealistic, navigable virtual reality experience". PR Newswire. OTOY. 2015-05-07. https://www.prnewswire.com/news-releases/otoy-demonstrates-first-ever-light-field-capture-of-a-real-world-environment-producing-photorealistic-navigable-virtual-reality-experience-300079502.html. Retrieved 2026-09-27.
- ↑ "OTOY demonstrates first ever light field capture for VR". YouTube. OTOY. https://www.youtube.com/watch?v=pyJUg-ja0cg. Retrieved 2026-09-27.
- ↑ 7.0 7.1 7.2 7.3 Will Mason (2015-11-04). "Lytro announces world's first light field VR video camera". UploadVR. https://www.uploadvr.com/lytro-immerge-vr-light-field-video-camera/. Retrieved 2026-09-27.
- ↑ 8.0 8.1 Marc Levoy, Pat Hanrahan (1996). "Light Field Rendering". Proceedings of SIGGRAPH 96 (ACM). pp. 31-42. doi:10.1145/237170.237199. https://graphics.stanford.edu/papers/light/. Retrieved 2026-09-27.
- ↑ E. H. Adelson, J. Y. A. Wang (1992). "Single lens stereo with a plenoptic camera". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2. pp. 99-106. https://doi.org/10.1109/34.121783. Retrieved 2026-09-27.
- ↑ 10.0 10.1 Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, Michael F. Cohen (August 1996). "The Lumigraph". Proceedings of SIGGRAPH 96 (ACM). Microsoft Research. pp. 43-54. doi:10.1145/237170.237200. https://www.microsoft.com/en-us/research/publication/the-lumigraph/. Retrieved 2026-09-27.
- ↑ Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (July 2005). "High Performance Imaging Using Large Camera Arrays". ACM Transactions on Graphics, vol. 24, no. 3 (SIGGRAPH 2005). pp. 765-776. doi:10.1145/1073204.1073259. https://graphics.stanford.edu/papers/CameraArray/. Retrieved 2026-09-27.
- ↑ 12.0 12.1 Ren Ng, Marc Levoy, Mathieu Bredif, Gene Duval, Mark Horowitz, Pat Hanrahan (2005). "Light Field Photography with a Hand-Held Plenoptic Camera". Stanford University Computer Science Tech Report CSTR 2005-02. https://graphics.stanford.edu/papers/lfcamera/. Retrieved 2026-09-27.
- ↑ 13.0 13.1 Megan Rose Dickey (2018-03-27). "Lytro, a light-field imaging startup, is shutting down". TechCrunch. https://techcrunch.com/2018/03/27/lytro-is-shutting-down. Retrieved 2026-09-27.
- ↑ 14.0 14.1 14.2 14.3 Shuyao Zhou, Tianqian Zhu, Kanle Shi, Yazi Li, Wen Zheng, Junhai Yong (December 2021). "Review of light field technologies". Visual Computing for Industry, Biomedicine, and Art, vol. 4, article 29. doi:10.1186/s42492-021-00096-8. https://doi.org/10.1186/s42492-021-00096-8. Retrieved 2026-09-27.
- ↑ 15.0 15.1 Ryan Damm (2015-05-07). "Still Life with Light Field: Explaining OTOY's Light Field Technology". UploadVR. https://www.uploadvr.com/still-life-with-light-field-explaining-otoys-light-field-tech/. Retrieved 2026-09-27.
- ↑ 16.0 16.1 16.2 Brittany Hillen (2017-11-29). "Lytro Immerge 2.0 is a HUGE light-field camera rig for high-end VR production". DPReview. https://www.dpreview.com/news/6149054684/lytro-immerge-2-0-is-a-huge-light-field-camera-rig-for-high-end-vr-production. Retrieved 2026-09-27.
- ↑ 17.0 17.1 17.2 Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, Paul Debevec (2020). "Immersive Light Field Video with a Layered Mesh Representation". ACM Transactions on Graphics, vol. 39, no. 4 (SIGGRAPH 2020). Google. doi:10.1145/3386569.3392485. https://augmentedperception.github.io/deepviewvideo/. Retrieved 2026-09-27.
- ↑ Daniel Cooper (2015-05-11). "We're one step closer to navigable 3D pictures of real places". Engadget. https://www.engadget.com/2015/05/11/otoy-light-field-vr. Retrieved 2026-09-27.
- ↑ Ben Lang (2016-08-30). "Lytro Shows First Light Field Footage Captured with Immerge VR Camera". Road to VR. https://www.roadtovr.com/lytro-shows-first-light-field-footage-captured-with-immerge-vr-camera/. Retrieved 2026-09-27.
- ↑ 20.0 20.1 Ben Lang (2017-11-27). "Exclusive: Lytro Reveals Immerge 2.0 Light-field Camera with Improved Quality, Faster Captures". Road to VR. https://roadtovr.com/exclusive-lytro-reveals-immerge-2-0-light-field-camera-improved-quality-faster-captures/. Retrieved 2026-09-27.
- ↑ Scott Hayden (2020-06-30). "Google Takes a Step Closer to Making Volumetric VR Video Streaming a Thing". Road to VR. https://www.roadtovr.com/google-volumetric-vr-video-siggraph-2020/. Retrieved 2026-09-27.
- ↑ Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar (2019). "Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines". ACM Transactions on Graphics (SIGGRAPH 2019). https://arxiv.org/abs/1905.00889. Retrieved 2026-09-27.
- ↑ Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng (2020). "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis". arXiv (ECCV 2020). https://arxiv.org/abs/2003.08934. Retrieved 2026-09-27.
- ↑ Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, George Drettakis (July 2023). "3D Gaussian Splatting for Real-Time Radiance Field Rendering". ACM Transactions on Graphics, vol. 42, no. 4. https://arxiv.org/abs/2308.04079. Retrieved 2026-09-27.