Toggle search
Toggle menu
4.9K
5.5K
229
46.8K
VR & AR Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Light field camera

From VR & AR Wiki
More actions

A light field camera, also called a plenoptic camera, is a camera that records the direction of incoming light rays as well as their intensity, so that it samples the 4D light field of a scene instead of a single flat image. In the microlens design described by Edward Adelson and John Wang in 1992 and built as a hand-held camera at Stanford University in 2005, a microlens array sits between the main lens and the image sensor; each microlens forms a small image of the main lens aperture on the pixels behind it, which records where on the aperture each ray came from.[1][2] Software can then compute photographs focused at different depths, shift the viewpoint slightly, or estimate depth, all from one exposure.[2][1] Arrays of many cameras can capture the same kind of data from more widely spaced viewpoints, and researchers also describe such rigs as light field cameras.[3][4][5]

Lytro shipped consumer plenoptic cameras from 2012 and later built very large light field rigs for cinematic virtual reality,[6][7] while the German company Raytrix has sold plenoptic cameras for professional and research use since 2010.[8] For VR the attraction is parallax: footage captured as a light field lets a viewer in a headset move their head within the captured volume and see the scene shift correctly, which standard 360-degree video cannot do.[7][9] The general process of recording light fields, including rotating rigs and camera domes, is covered in light field capture; devices that reproduce light fields for the eye are covered in light field display.

Reviewed 4 October 2026. Optics, resolution figures, history, Lytro, Raytrix and Google rig specifications checked against the cited papers (DOIs via Crossref) and articles. About review dates.

How it works

Plenoptic design

Adelson and Wang's paper describes a single main lens with a "lenticular array" placed at the sensor plane. They compare the microlens array to "a battery of tiny cameras, where each tiny camera forms an image of the main lens aperture as seen from a different position in the sensor plane." Each lenslet gathers light over one "macropixel" of the sensor, and the recorded data shows how the world appears from viewpoints within the lens aperture. If a macropixel is divided into n subpixels, addressing different subpixels gives n views from slightly different positions.[1] Their stated goal was "single lens stereo": estimating object depth from the parallax between these views, using both horizontal and vertical parallax; they wrote that the correspondence problem of two-camera stereo is minimized.[1]

Ng and colleagues explain that each microlens measures not only the total light arriving at its position but "how much light arrives along each ray". By re-sorting the measured rays to where they would have ended in a slightly different, synthetic cameras, software can compute sharp photographs focused at different depths. The paper reports that a linear increase in the resolution of the images under each microlens gives a linear increase in the sharpness of refocused photographs, so the depth of field can be extended without closing the aperture.[2]

Spatial and angular resolution

A plenoptic camera divides its sensor pixels between position and direction. Adelson and Wang note that if each macropixel is divided into n subpixels along one dimension, spatial resolution falls by a factor of n in that dimension; with 5 x 5 macropixels, a 500 x 500 element sensor gives a spatial resolution of only 100 x 100.[1] Levoy's 2006 survey states the general rule: with P x P microlenses and N x N pixels under each, computed views have P x P pixels, and a camera with an f/A main lens can refocus anywhere within the depth of field of an f/(A x N) lens. His example used a 16-megapixel sensor, an f/4 main lens and 300 x 300 microlenses, which gave refocusing over the depth of field of an f/56 camera but output images of only 300 x 300 pixels.[3] The range of available viewpoints is limited by the diameter of the main lens aperture, so the technique works best in macrophotography, where the scene is close to the camera and therefore large relative to the aperture.[3]

Raytrix describes the same trade-off for its industrial cameras: their maximum effective lateral resolution is about one quarter of the sensor resolution, it varies over the depth of field, and depth resolution is typically about 1% of the total depth of field. The company notes that a wide-angle main lens can only resolve depth differences close to the camera.[10]

Standard and focused plenoptic cameras

In the original or "unfocused" design (sometimes called plenoptic camera 1.0), the main lens is focused on the microlens array, the microlenses are focused at infinity, and the sensor sits at their focal plane. Spatial resolution at the focal plane then equals the number of microlenses.[11][12] In 2009 Andrew Lumsdaine and Todor Georgiev introduced the focused plenoptic camera, also called plenoptic camera 2.0.[13][11] Here the microlenses are focused on the image formed by the main lens, in front of or behind the array, so each microlens produces a focused image. This shifts the balance toward spatial resolution and samples ray direction more loosely.[11][12] Raytrix cameras use the focused design, and calibration research treats the two types separately.[11][12]

Camera arrays and other designs

A light field can also be captured with many separate cameras. In 2005 Bennett Wilburn, Marc Levoy and colleagues at Stanford described an array of 100 custom video cameras, used for applications such as synthetic aperture video and spatiotemporal view interpolation.[14] Levoy notes that long-baseline light fields of a dynamic scene need multiple cameras, while for a short baseline (inches to microns) a single camera with an array of lenses can replace them.[3] At the other end of the size range, Pelican Imaging's PiCam, presented in 2013, was a monolithic camera array small enough for smartphones that captured light fields and computed high-resolution images along with a range image of scene depth.[15]

A 2021 review by Zhou and colleagues summarizes an earlier survey by Wu et al. that sorts acquisition methods into multisensor capture (mostly camera arrays), time-sequential capture with one camera over several exposures, and multiplexed imaging that encodes the higher-dimensional data in one 2D image; the review calls multiplexed imaging the most popular method.[16] The same review notes that, unlike the Lytro camera with its microlens array, Wooptix reduced the resolution trade-off by placing a liquid lens in front of the sensor to change focal planes quickly, providing the full resolution of the sensor in real time.[16]

History

The use of lens arrays to record light fields goes back to Gabriel Lippmann's 1908 invention of integral photography.[3] Adelson and Wang pointed out that their camera used optical principles laid out by Lippmann and by Herbert E. Ives in early experiments on 3D photography; they cite Ives's 1930 paper on parallax panoramagrams made with a large-diameter lens.[1] Their 1992 paper took the name "plenoptic camera" from the plenoptic function, the term Adelson and James Bergen used for the intensity of each light ray as a function of visual angle, wavelength, time and viewing position; "plenoptic" comes from roots meaning "complete" and "view".[1] According to Ng and colleagues, Adelson and Wang did not build a portable version; their prototype used a relay lens to form the image on a separate sensor.[1][2]

In April 2005 Ren Ng, Marc Levoy, Mathieu Bredif, Gene Duval, Mark Horowitz and Pat Hanrahan published a hand-held plenoptic camera that "looks and operates exactly like a conventional camera" from outside.[2] The prototype used a Contax 645 medium-format body with a Megavision FB4040 digital back, whose Kodak KAF-16802CE sensor had about 4000 x 4000 pixels, 9 microns wide. A microlens array from Adaptive Optics Associates with 296 x 296 square lenslets, each 125 microns wide with a focal length of 500 microns (f/4), was mounted in front of the sensor. The resulting light fields measured 292 x 292 in spatial resolution and just under 14 x 14 in directional resolution.[2] Ng completed his Stanford doctorate in 2006 with the dissertation "Digital Light Field Photography", which earned best-dissertation recognition from the Association for Computing Machinery, and founded Lytro after graduating.[6] In 2006 Levoy's group also inserted a microlens array into a conventional microscope to capture light fields of biological specimens in a single photograph.[17]

Raytrix was founded in Kiel, Germany, in 2008 and has sold 3D light field cameras for professional and research use since 2010.[8] Raytrix lists among its technical publications the paper "Single Lens 3D-Camera with Extended Depth-of-Field", which Christian Perwass and Lennart Wietzke published in the Proceedings of SPIE in 2012.[10][18] Lytro announced its first consumer camera on 19 October 2011 capturing 11 megarays of light field data, with an 8x optical zoom and an f/2 lens, priced at US$399 (8 GB) and US$499 (16 GB); it shipped in late February 2012.[19][6] Lytro later moved into VR and cinema capture,[7][20] and on 27 March 2018 said it would begin winding down; some of its team joined Google.[21]

Commercial and production systems

System Maker Announced Design Notes
Lytro Light Field Camera Lytro 19 October 2011 Microlens plenoptic 11-megaray sensor, 8x zoom, f/2 lens; US$399 (8 GB) or US$499 (16 GB)[19]
Lytro Illum Lytro 22 April 2014 Microlens plenoptic 40-megaray sensor, 30-250 mm f/2 lens, 4-inch touchscreen, Qualcomm Snapdragon 800; US$1,599, available 15 July 2014[22]
Lytro Immerge Lytro 4 November 2015 Camera array with server Modular rings of cameras; six degrees of freedom inside a capture volume of about one meter[7]
Lytro Cinema Lytro April 2016 Light field cinema camera 755 raw megapixels, up to 300 fps, up to 400 GB per second of data; rental packages from US$125,000[20]
Lytro Immerge 2.0 Lytro November 2017 Camera array 95 cameras in alternating rows; 120 degrees per position, three rotations for 360 degrees[23]
Raytrix cameras Raytrix Sold since 2010 Focused plenoptic Industrial and research cameras; raw images processed on a PC GPU into 2D images and depth[8][10][11]

Applications in VR and AR

Light field cameras matter to VR because a captured light field supports head motion. UploadVR described Lytro Immerge, announced in November 2015, as giving six degrees of freedom of movement within a capture volume of about one meter, with horizontal and vertical parallax.[7] Its successor, Immerge 2.0, used 95 cameras arranged so that each position covered 120 degrees instead of 90, cutting the rotations needed for a full 360-degree scene from five to three.[23]

Google's VR team took a related approach for still scenes. In March 2018 Paul Debevec described bending a GoPro Odyssey Jump camera into a vertical arc of 16 cameras on a rotating platform, which recorded about 1,000 outward-facing viewpoints on a 70 cm sphere in about a minute. The results were released as the free app Welcome to Light Fields on Steam for HTC Vive, Oculus Rift and Windows Mixed Reality headsets.[9] The accompanying SIGGRAPH Asia 2018 paper describes two light field camera rigs, a real-time renderer producing stereo views at 90 Hz on commodity VR hardware, and compression based on the VP9 video codec; it reports that the app had been downloaded more than 15,000 times.[5] For video, Google researchers presented at SIGGRAPH 2020 a hemispherical dome, 92 cm in diameter, carrying 46 time-synchronized cameras, with an 80 cm viewing baseline, 10 pixels per degree, a field of view above 220 degrees and 30 frames per second.[24] These rigs spread their cameras over a sphere or dome tens of centimeters across, while the viewpoints of a single microlens camera are limited by the diameter of its main lens aperture; the Google paper notes that lenslet arrays "typically capture too small a viewing volume for VR".[9][24][3][5] Zhou and colleagues write that 3D reconstruction from light fields gives researchers "a bright outlook for real-time augmented reality and virtual reality".[16]

Plenoptic cameras also produce depth from a single lens. Strobl and Lingenauber list passive 3D video recording, 3D modeling, range-based segmentation and tracking among potential uses, most of which rely on metric depth information from calibrated cameras.[11] The LiFCal method (2024) calibrates a focused plenoptic camera from a moving image sequence without a calibration target and demonstrates the result in depth estimation and SLAM.[12]

Other applications

Raytrix lists volumetric velocimetry, plant phenotyping, automated optical inspection and microscopy among the uses of its cameras, and says one camera can obtain 3D data through a standard microscope in a single shot.[8][10] Light field microscopy, introduced by Levoy's group in 2006, produces perspective views and focal stacks of a specimen from one photograph.[17] In cinema, Lytro presented its Cinema system as allowing depth of field, focus position, shutter speed and dynamic range to be changed after filming, and as a "depth screen" alternative to green screens.[20]

Data and standards

The JPEG committee's JPEG Pleno work covers light field coding. The committee describes light fields as typically produced either by an array of cameras or by "a single and more compact sensor augmented by microlenses".[4] Part 2 of the standard, ISO/IEC 21794-2:2021 "Light field coding", is published, with an amendment for profiles and levels; the committee's work plan lists a second edition at the final draft (FDIS) stage.[25]

See also

References

  1. ↑ 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 Edward H. Adelson, John Y.A. Wang (February 1992). "Single Lens Stereo with a Plenoptic Camera". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 99-106. doi:10.1109/34.121783. https://doi.org/10.1109/34.121783. Retrieved 2026-10-04.
  2. ↑ 2.0 2.1 2.2 2.3 2.4 2.5 Ren Ng, Marc Levoy, Mathieu Bredif, Gene Duval, Mark Horowitz, Pat Hanrahan (April 2005). "Light Field Photography with a Hand-Held Plenoptic Camera". Stanford University Computer Science Tech Report CSTR 2005-02. https://graphics.stanford.edu/papers/lfcamera/. Retrieved 2026-10-04.
  3. ↑ 3.0 3.1 3.2 3.3 3.4 3.5 Marc Levoy (August 2006). "Light Fields and Computational Imaging". Computer (IEEE Computer Society), vol. 39, no. 8, pp. 46-55. doi:10.1109/MC.2006.270. https://graphics.stanford.edu/papers/lfphoto/levoy-lfphoto-ieee06.pdf. Retrieved 2026-10-04.
  4. ↑ 4.0 4.1 "JPEG Pleno Light Field". JPEG. ISO/IEC JTC 1/SC 29/WG 1. https://jpeg.org/jpegpleno/lightfield.html. Retrieved 2026-10-04.
  5. ↑ 5.0 5.1 5.2 Ryan S. Overbeck, Daniel Erickson, Daniel Evangelakos, Matt Pharr, Paul Debevec (2018). "A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality". ACM Transactions on Graphics, vol. 37, no. 6 (SIGGRAPH Asia 2018). doi:10.1145/3272127.3275031. https://arxiv.org/abs/1810.08860. Retrieved 2026-10-04.
  6. ↑ 6.0 6.1 6.2 Andrew Myers (2012-04-03). "Lytro's new perspective: Stanford dissertation leads to a photographic revolution". Stanford University School of Engineering. https://engineering.stanford.edu/news/lytros-new-perspective-stanford-dissertation-leads-photographic-revolution. Retrieved 2026-10-04.
  7. ↑ 7.0 7.1 7.2 7.3 7.4 Will Mason (2015-11-04). "Lytro announces world's first light field VR video camera". UploadVR. https://www.uploadvr.com/lytro-immerge-vr-light-field-video-camera/. Retrieved 2026-10-04.
  8. ↑ 8.0 8.1 8.2 8.3 "Company". Raytrix. raytrix GmbH. https://www.raytrix.de/company/. Retrieved 2026-10-04.
  9. ↑ 9.0 9.1 9.2 Paul Debevec (2018-03-14). "Experimenting with Light Fields". The Keyword (Google blog). Google. https://blog.google/products-and-platforms/products/google-ar-vr/experimenting-light-fields/. Retrieved 2026-10-04.
  10. ↑ 10.0 10.1 10.2 10.3 "3D light field technology". Raytrix. raytrix GmbH. https://www.raytrix.de/technology/. Retrieved 2026-10-04.
  11. ↑ 11.0 11.1 11.2 11.3 11.4 11.5 Klaus H. Strobl, Martin Lingenauber (2016). "Stepwise calibration of focused plenoptic cameras". Computer Vision and Image Understanding, vol. 145, pp. 140-147. doi:10.1016/j.cviu.2015.12.010. https://elib.dlr.de/103337/2/strobl16cviu.pdf. Retrieved 2026-10-04.
  12. ↑ 12.0 12.1 12.2 12.3 Aymeric Fleith, Doaa Ahmed, Daniel Cremers, Niclas Zeller (2024-08-21). "LiFCal: Online Light Field Camera Calibration via Bundle Adjustment". German Conference on Pattern Recognition (GCPR) 2024, arXiv preprint. https://arxiv.org/abs/2408.11682. Retrieved 2026-10-04.
  13. ↑ Andrew Lumsdaine, Todor Georgiev (April 2009). "The focused plenoptic camera". 2009 IEEE International Conference on Computational Photography (ICCP). pp. 1-8. doi:10.1109/ICCPHOT.2009.5559008. https://doi.org/10.1109/ICCPHOT.2009.5559008. Retrieved 2026-10-04.
  14. ↑ Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (July 2005). "High performance imaging using large camera arrays". ACM Transactions on Graphics, vol. 24, no. 3, pp. 765-776 (SIGGRAPH 2005). doi:10.1145/1073204.1073259. https://doi.org/10.1145/1073204.1073259. Retrieved 2026-10-04.
  15. ↑ Kartik Venkataraman, Dan Lelescu, Jacques Duparre, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, Shree Nayar (November 2013). "PiCam: an ultra-thin high performance monolithic camera array". ACM Transactions on Graphics, vol. 32, no. 6 (SIGGRAPH Asia 2013). doi:10.1145/2508363.2508390. https://doi.org/10.1145/2508363.2508390. Retrieved 2026-10-04.
  16. ↑ 16.0 16.1 16.2 Shuyao Zhou, Tianqian Zhu, Kanle Shi, Yazi Li, Wen Zheng, Junhai Yong (December 2021). "Review of light field technologies". Visual Computing for Industry, Biomedicine, and Art, vol. 4, article 29. doi:10.1186/s42492-021-00096-8. https://doi.org/10.1186/s42492-021-00096-8. Retrieved 2026-10-04.
  17. ↑ 17.0 17.1 Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, Mark Horowitz (July 2006). "Light Field Microscopy". ACM Transactions on Graphics, vol. 25, no. 3, pp. 924-934 (SIGGRAPH 2006). doi:10.1145/1141911.1141976. https://graphics.stanford.edu/papers/lfmicroscope/. Retrieved 2026-10-04.
  18. ↑ Christian Perwass, Lennart Wietzke (2012). "Single lens 3D-camera with extended depth-of-field". Human Vision and Electronic Imaging XVII, Proceedings of SPIE vol. 8291. doi:10.1117/12.909882. https://doi.org/10.1117/12.909882. Retrieved 2026-10-04.
  19. ↑ 19.0 19.1 Dante Cesa (2011-10-19). "Lytro introduces world's first light field camera: f/2 lens, $399, ships early 2012". Engadget. https://www.engadget.com/2011-10-19-lytro-introduces-worlds-first-lightfield-camera.html. Retrieved 2026-10-04.
  20. ↑ 20.0 20.1 20.2 Lucas Matney (2016-04-11). "Lytro's 755 megapixel Cinema light field camera is going to kill the green screen". TechCrunch. https://techcrunch.com/2016/04/11/lytro-cinema-is-giving-filmmakers-400-gigabytes-per-second-of-creative-freedom. Retrieved 2026-10-04.
  21. ↑ Megan Rose Dickey (2018-03-27). "Lytro, a light-field imaging startup, is shutting down". TechCrunch. https://techcrunch.com/2018/03/27/lytro-is-shutting-down/. Retrieved 2026-10-04.
  22. ↑ Nicole Lee (2014-04-22). "Lytro's new light-field camera looks like an actual camera, costs $1,599". Engadget. https://www.engadget.com/2014-04-22-lytro-illum.html. Retrieved 2026-10-04.
  23. ↑ 23.0 23.1 Brittany Hillen (2017-11-29). "Lytro Immerge 2.0 is a HUGE light-field camera rig for high-end VR production". DPReview. https://www.dpreview.com/news/6149054684/lytro-immerge-2-0-is-a-huge-light-field-camera-rig-for-high-end-vr-production. Retrieved 2026-10-04.
  24. ↑ 24.0 24.1 Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, Paul Debevec (2020). "Immersive Light Field Video with a Layered Mesh Representation". ACM Transactions on Graphics, vol. 39, no. 4 (SIGGRAPH 2020). Google. doi:10.1145/3386569.3392485. https://augmentedperception.github.io/deepviewvideo/. Retrieved 2026-10-04.
  25. ↑ "JPEG Pleno workplan". JPEG. ISO/IEC JTC 1/SC 29/WG 1. https://jpeg.org/jpegpleno/workplan.html. Retrieved 2026-10-04.
Retrieved from "https://vrarwiki.com/index.php?title=Light_field_camera&oldid=46437"