Jump to content

Light field: Difference between revisions

From VR & AR Wiki
Newer edit →
VisualWikitext
Created page with "{{DISPLAYTITLE:Light Field}} A '''light field''' (also spelled '''lightfield''') is a fundamental concept in optics and computer graphics that describes the amount of light traveling in every direction through every point in space.<ref name="LevoyHanrahan1996">Levoy, M., & Hanrahan, P. (1996). Light field rendering. ''Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96'', 31–42.</ref><ref name="Gortle..."
 
No edit summary
Line 1: Line 1:
{{DISPLAYTITLE:Light Field}}
{{DISPLAYTITLE:Light Field}}
A '''light field''' (also spelled '''lightfield''') is a fundamental concept in [[optics]] and [[computer graphics]] that describes the amount of [[light]] traveling in every direction through every point in [[space]].<ref name="LevoyHanrahan1996">Levoy, M., & Hanrahan, P. (1996). Light field rendering. ''Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96'', 31–42.</ref><ref name="Gortler1996">Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996). The Lumigraph. ''Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96'', 43–54.</ref> Essentially, it's a function that represents the radiance of light rays at any position and direction within a given volume or area. Understanding and utilizing light fields is crucial for advancing [[virtual reality]] (VR) and [[augmented reality]] (AR) technologies, as it allows for the capture and reproduction of visual scenes with unprecedented realism, including effects like [[parallax]] and [[refocusing]] after capture.<ref name="Ng2005">Ng, R. (2005). Digital Light Field Photography. ''Ph.D. Thesis, Stanford University''.</ref>
A '''light field''' (also spelled '''lightfield''') is a fundamental concept in [[optics]] and [[computer graphics]] that describes the amount of [[light]] traveling in every direction through every point in [[space]].<ref name="LevoyHanrahan1996">Levoy, M., & Hanrahan, P. (1996). Light field rendering. ''Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96'', 31–42.</ref><ref name="Gortler1996">Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996). The Lumigraph. ''Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96'', 43–54.</ref> Essentially, it's a vector function that represents the [[radiance]] of light rays at any position and direction within a given volume or area. Understanding and utilizing light fields is crucial for advancing [[virtual reality]] (VR) and [[augmented reality]] (AR) technologies, as it allows for the capture and reproduction of visual scenes with unprecedented realism, including effects like [[parallax]], [[reflection]]s, [[refraction]]s, and [[refocusing]] after capture, while also aiming to solve critical issues like the [[vergence-accommodation conflict]].<ref name="Ng2005">Ng, R. (2005). Digital Light Field Photography. ''Ph.D. Thesis, Stanford University''.</ref><ref name="Lanman2013">Lanman, D., & Luebke, D. (2013). Near-eye light field displays. ''ACM SIGGRAPH 2013 Talks'', 1-1.</ref>


==History==
==History==
The concept of measuring light rays has early roots. [[Michael Faraday]] first speculated in 1846 in his lecture "Thoughts on Ray Vibrations" that light should be understood as a field, similar to the [[magnetic field]] he had studied.<ref name="Faraday1846">Faraday, M. (1846). Thoughts on Ray Vibrations. ''Philosophical Magazine'', S.3, Vol. 28, No. 188.</ref> The term "light field" (''svetovoe pole'' in Russian) was more formally defined by [[Andrey Gershun]] in a classic 1936 paper on the radiometric properties of light in three-dimensional space.<ref name="Gershun1936">Gershun, A. (1939). The Light Field. ''Journal of Mathematics and Physics'', 18(1-4), 51–151. (English translation of 1936 Russian paper).</ref>
The concept of measuring light rays has early roots. [[Michael Faraday]] first speculated in 1846 in his lecture "Thoughts on Ray Vibrations" that light should be understood as a field, similar to the [[magnetic field]] he had studied.<ref name="Faraday1846">Faraday, M. (1846). Thoughts on Ray Vibrations. ''Philosophical Magazine'', S.3, Vol. 28, No. 188.</ref> The term "light field" (''svetovoe pole'' in Russian) was more formally defined by [[Andrey Gershun]] in a classic 1936 paper on the radiometric properties of light in three-dimensional space.<ref name="Gershun1936">Gershun, A. (1939). The Light Field. ''Journal of Mathematics and Physics'', 18(1-4), 51–151. (English translation of 1936 Russian paper).</ref><ref name="WikiLF">[https://en.wikipedia.org/wiki/Light_field Wikipedia: Light field]</ref>


In the context of [[computer vision]] and graphics, the concept was further developed with the introduction of the 7D [[plenoptic function]] by [[Edward Adelson|Adelson]] and [[James Bergen|Bergen]] in 1991.<ref name="AdelsonBergen1991">Adelson, E. H., & Bergen, J. R. (1991). The plenoptic function and the elements of early vision. In ''Computational Models of Visual Processing'' (pp. 3-20). MIT Press.</ref> This function describes all possible light rays, parameterized by 3D position (x, y, z), 2D direction (θ, φ), wavelength (λ), and time (t).
In the context of [[computer vision]] and graphics, the concept was further developed with the introduction of the 7D [[plenoptic function]] by [[Edward Adelson|Adelson]] and [[James Bergen|Bergen]] in 1991.<ref name="AdelsonBergen1991">Adelson, E. H., & Bergen, J. R. (1991). The plenoptic function and the elements of early vision. In ''Computational Models of Visual Processing'' (pp. 3-20). MIT Press.</ref> This function describes all possible light rays, parameterized by 3D position (x, y, z), 2D direction (θ, φ), wavelength (λ), and time (t).


Practical computational approaches often reduce the dimensionality. Two seminal papers in 1996, "Light Field Rendering" by Levoy and Hanrahan<ref name="LevoyHanrahan1996"/> and "The Lumigraph" by Gortler et al.<ref name="Gortler1996"/>, independently proposed using a 4D subset of the plenoptic function for capturing and rendering complex scenes. They introduced the highly influential [[two-plane parameterization]] (2PP), simplifying the representation by defining rays based on their intersection points with two parallel planes. This 4D light field representation forms the basis for most modern light field capture and display technologies.
Practical computational approaches often reduce the dimensionality. Two seminal papers in 1996, "Light Field Rendering" by Levoy and Hanrahan at [[Stanford University]]<ref name="LevoyHanrahan1996"/> and "The Lumigraph" by Gortler et al.<ref name="Gortler1996"/>, independently proposed using a 4D subset of the plenoptic function for capturing and rendering complex scenes without detailed 3D models.<ref name="StanfordLF">[http://graphics.stanford.edu/projects/lightfield/ Stanford University: Light fields and computational photography]</ref> They introduced the highly influential [[two-plane parameterization]] (2PP), simplifying the representation by defining rays based on their intersection points with two parallel planes. This 4D light field representation forms the basis for most modern light field capture and display technologies. The introduction of [[light field camera]]s, inspired by Gabriel Lippmann’s 1908 concept of [[integral photography]], further advanced the field.<ref name="WikiLFCam">[https://en.wikipedia.org/wiki/Light_field_camera Wikipedia: Light field camera]</ref>


==Theory and Representation==
==Theory and Representation==
Line 13: Line 13:


===The Plenoptic Function===
===The Plenoptic Function===
The most complete representation is the 7D plenoptic function, P(x, y, z, θ, φ, λ, t), describing the [[radiance]] of light at any 3D point (x,y,z), in any direction (θ, φ), for any wavelength (λ), at any time (t).<ref name="AdelsonBergen1991"/> For many applications, this is overly complex and contains redundant information (e.g., light doesn't typically change along a straight ray in free space unless wavelength or time are critical).
The most complete representation is the 7D plenoptic function, P(x, y, z, θ, φ, λ, t), describing the radiance of light at any 3D point (x,y,z), in any direction (θ, φ), for any wavelength (λ), at any time (t).<ref name="AdelsonBergen1991"/> For many applications, this is overly complex and contains redundant information (e.g., light doesn't typically change along a straight ray in free space—radiance invariance—unless wavelength or time are critical).<ref name="WikiLF"/>


===4D Light Field===
===Simplified Light Fields===
For static scenes under constant illumination, the time (t) and wavelength (λ, often simplified to [[RGB]] channels) dependencies can often be dropped. Furthermore, due to the constancy of radiance along a ray in free space, the 3D spatial component can be reduced. The most common simplification is the 4D light field.<ref name="LevoyHanrahan1996"/><ref name="Gortler1996"/>
For static scenes under constant illumination, the time (t) and wavelength (λ, often simplified to [[RGB]] channels) dependencies can often be dropped. Furthermore, due to the constancy of radiance along a ray in free space, the 3D spatial component can be reduced.
* '''5D Light Field:''' Often represented as L = L(x, y, z, θ, φ).<ref name="Art1Faraday">Faraday, M. (1846). "Experimental Researches in Electricity." Philosophical Transactions of the Royal Society of London, 136, 1-20.</ref> <!-- Note: Article 1 incorrectly cites Faraday for 5D LF; using the reference marker as requested but the citation itself is questionable for this specific point -->
* '''4D Light Field:''' The most common simplification, often called the [[photic field]] or [[lumigraph]] in regions free of occluders.<ref name="WikiLF"/> It captures radiance along rays without redundant data.<ref name="LevoyHanrahan1996"/><ref name="Gortler1996"/>


====Two-Plane Parameterization (2PP)====
====Two-Plane Parameterization (2PP)====
Line 28: Line 30:
Capturing a light field involves sampling the intensity and direction of light rays within a scene. Several methods exist:
Capturing a light field involves sampling the intensity and direction of light rays within a scene. Several methods exist:


* '''[[Light Field Camera]]s (Plenoptic Cameras):''' These are the most common devices. They typically insert a microlens array between the main lens and the [[image sensor]].<ref name="AdelsonWang1992">Adelson, E. H., & Wang, J. Y. A. (1992). Single lens stereo with a plenoptic camera. ''IEEE Transactions on Pattern Analysis and Machine Intelligence'', 14(2), 99-106.</ref><ref name="Ng2005"/> Each microlens samples the light arriving from the main lens from slightly different perspectives, directing these samples onto different [[pixel]]s on the sensor below it. The sensor thus records not only the total light hitting each microlens (spatial information) but also how that light is distributed directionally (angular information). Consumer examples included cameras from [[Lytro]] (now defunct) and Raytrix.
* '''[[Light Field Camera]]s (Plenoptic Cameras):''' These are common devices that typically insert a microlens array between the main lens and the [[image sensor]].<ref name="AdelsonWang1992">Adelson, E. H., & Wang, J. Y. A. (1992). Single lens stereo with a plenoptic camera. ''IEEE Transactions on Pattern Analysis and Machine Intelligence'', 14(2), 99-106.</ref><ref name="Ng2005"/> Each microlens samples the light arriving from the main lens from slightly different perspectives, directing these samples onto different [[pixel]]s on the sensor below it. The sensor thus records not only spatial information but also angular information.<ref name="WikiLFCam"/> Consumer examples included cameras from [[Lytro]] (now defunct) and Raytrix.<ref name="Ng2005"/>
* '''Camera Arrays:''' A synchronized array of conventional cameras can capture a light field.<ref name="Wilburn2005">Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., ... & Levoy, M. (2005). High performance imaging using large camera arrays. ''ACM Transactions on Graphics (TOG)'', 24(3), 765-776.</ref> Each camera samples the scene from a different viewpoint. By combining the images and knowing the cameras' precise positions and orientations, the 4D light field can be reconstructed. This approach often yields higher spatial resolution than single plenoptic cameras but requires careful calibration and synchronization.
* '''Camera Arrays:''' A synchronized array of conventional cameras can capture a light field by recording the scene from slightly different viewpoints simultaneously.<ref name="Wilburn2005">Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., ... & Levoy, M. (2005). High performance imaging using large camera arrays. ''ACM Transactions on Graphics (TOG)'', 24(3), 765-776.</ref> This approach often yields higher spatial resolution but requires careful calibration and synchronization.<ref name="Art1Wilburn">Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., Adams, A., Horowitz, M., & Levoy, M. (2005). "High performance imaging using large camera arrays." ACM Transactions on Graphics, 24(3), 765-776.</ref>
* '''Scanning/Gantry Setups:''' A single camera moved precisely to multiple positions can sequentially capture the views needed to sample the light field. This is suitable for static scenes.
* '''Scanning/Gantry Setups:''' A single camera moved precisely to multiple positions can sequentially capture the views needed to sample the light field. This is suitable for static scenes but slower.<ref name="Gortler1996"/>
* '''[[Coded Aperture]] Photography:''' Placing a patterned mask (coded aperture) near the sensor or aperture can modulate incoming light in a way that allows directional information to be computationally recovered.<ref name="Veeraraghavan2007">Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007). Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ''ACM Transactions on Graphics (TOG)'', 26(3), 69-es.</ref>
* '''[[Coded Aperture]] Photography:''' Placing a patterned mask (coded aperture) near the sensor or aperture can modulate incoming light in a way that allows directional information to be computationally recovered.<ref name="Veeraraghavan2007">Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007). Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ''ACM Transactions on Graphics (TOG)'', 26(3), 69-es.</ref>
* '''Synthetic Light Fields:''' In [[computer-generated imagery]] (CGI), light fields can be generated synthetically by rendering a scene from multiple viewpoints. This is particularly relevant for VR and AR where scenes might be entirely or partially virtual.<ref name="Art1Moon">Moon, B., McDermott, S., Golas, K., Cao, Y., & Wong, T.-T. (2016). "Efficient rendering of virtual objects for mixed reality." Computers & Graphics, 57, 93-101.</ref>
* '''[[Computational Photography]] Techniques:''' Various other methods combining optics and computation are continuously being developed.
* '''[[Computational Photography]] Techniques:''' Various other methods combining optics and computation are continuously being developed.


The raw data captured by these methods needs significant processing to reconstruct the 4D light field representation, L(u, v, s, t).
==Light Field Processing==
Raw light field data requires significant processing:
 
* '''Calibration and Rectification:''' Raw data needs calibration to correct for lens distortions, sensor noise, and camera misalignments, especially in camera arrays or plenoptic systems. Geometric calibration ensures accurate reconstruction.<ref name="Art1Bok">Bok, Y., Jeon, H. G., & Kweon, I. S. (2017). "Geometric calibration of micro-lens-based light field cameras using line features." IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(2), 287-300.</ref>
* '''Interpolation and View Synthesis:''' A key advantage is generating novel viewpoints not explicitly captured. This involves interpolating the 4D light field data to estimate the scene's appearance from arbitrary positions and angles.<ref name="Art1Kalantari">Kalantari, N. K., Wang, T. C., & Ramamoorthi, R. (2016). "Learning-based view synthesis for light field cameras." ACM Transactions on Graphics, 35(6), 193.</ref>
* '''[[Depth Estimation]]:''' The angular variation of light rays encodes depth information. Various algorithms can extract depth maps, valuable for effects like synthetic [[depth of field]] and for AR interactions.<ref name="Art1Tao">Tao, M. W., Hadap, S., Malik, J., & Ramamoorthi, R. (2013). "Depth from combining defocus and correspondence using light-field cameras." Proceedings of the IEEE International Conference on Computer Vision, 673-680.</ref>
* '''Compression:''' Light field datasets are massive. Efficient compression is vital for storage and transmission, especially for mobile VR/AR and streaming. Techniques often adapt existing video codecs (like VP9) or use specialized approaches.<ref name="Art1Viola">Viola, I., Rerabek, M., & Ebrahimi, T. (2017). "Comparison and evaluation of light field image coding approaches." IEEE Journal of Selected Topics in Signal Processing, 11(7), 1092-1106.</ref><ref name="Art2AugPerc">[https://augmentedperception.github.io/welcome-to-lightfields/ Augmented Perception: Welcome to Light Fields]</ref> Standards bodies like JPEG Pleno and MPEG Immersive Video are developing formats for light field data.<ref name="Art4MMCommSoc">[https://mmc.committees.comsoc.org/files/2017/11/MMTC-Review-Letter-Vol-8-No-2-Nov-2017.pdf IEEE ComSoc MMTC Review Letter, Vol. 8, No. 2, Nov 2017]</ref> <!-- Rough citation combining refs from Art 4 -->


==Light Field Rendering and Display==
==Light Field Rendering and Display==
Once a light field is captured or synthetically generated, it can be used to render new views of the scene.
Once captured or generated, light fields enable novel rendering and display possibilities.


===Rendering===
===Rendering===
Rendering novel views involves sampling the 4D light field data. For a desired virtual camera position and orientation, the rendering algorithm calculates which rays from the light field would reach the virtual camera's sensor plane and integrates their radiance values.<ref name="LevoyHanrahan1996"/> This allows for:
Rendering involves sampling the 4D light field data to synthesize a 2D image for a virtual camera. For a desired camera pose, the algorithm determines which rays intersect the virtual sensor and integrates their values.<ref name="LevoyHanrahan1996"/> This enables:
* '''[[Refocusing]]:''' Shifting the virtual focal plane after capture by appropriately integrating rays.
* '''Refocusing:''' Synthetically changing the focal plane after capture.<ref name="Ng2005"/>
* '''Changing [[Depth of Field]] (DoF):''' Adjusting the aperture size computationally.
* '''Depth of Field Control:''' Computationally adjusting the aperture size.
* '''Small Viewpoint Shifts:''' Generating views from slightly different positions than the original capture positions, enabling parallax effects.
* '''View Synthesis:''' Generating views from slightly different positions, enabling parallax effects and [[six degrees of freedom]] (6DoF) movement.<ref name="Lanman2013"/>
* '''[[Neural Radiance Fields]] (NeRF) and Neural Rendering:''' Modern approaches like NeRF represent the scene as a continuous function learned by a [[neural network]], allowing high-quality view synthesis from sparse inputs. These methods effectively learn a compressed light field representation and are highly relevant for VR/AR rendering.<ref name="Art1Mildenhall">Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). "NeRF: Representing scenes as neural radiance fields for view synthesis." European Conference on Computer Vision, 405-421.</ref><ref name="Art4VRNeRF">[https://vr-nerf.github.io/ VR-NeRF Project Page]</ref>


===Light Field Displays===
===Light Field Displays===
Displaying light fields aims to reproduce the captured directional light rays, allowing the viewer to perceive depth and parallax naturally by moving their head. This is a key area of research for future VR and AR [[head-mounted display]]s (HMDs). Approaches include:
Light field displays aim to reproduce the directional aspect of light rays, allowing the viewer's eyes to [[accommodation|accommodate]] (focus) naturally at different depths, potentially resolving the vergence-accommodation conflict inherent in conventional stereoscopic displays.<ref name="Art1Huang">Huang, F. C., Chen, K., & Wetzstein, G. (2015). "The light field stereoscope: Immersive computer graphics via factored near-eye light field displays with focus cues." ACM Transactions on Graphics, 34(4), 60.</ref><ref name="Konrad2017">Konrad, R., Cooper, E. A., Wetzstein, G., & Banks, M. S. (2017). Accommodation and vergence responses to near-eye light field displays. ''Journal of Vision'', 17(10), 987-987.</ref> This provides more natural depth cues and improves visual comfort. Key approaches include:
* '''Multi-layer Displays:''' Using stacked [[LCD]] or [[OLED]] panels with attenuating layers to sculpt the light directionally.<ref name="Wetzstein2011">Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2011). Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays. ''ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2011'', 30(4), 95:1-95:12.</ref>
 
* '''Microlens Array Displays:''' Essentially the reverse of a plenoptic camera, where a display panel (e.g., OLED) emits light through a microlens array to project different images in different directions.<ref name="Jones2007">Jones, A., McDowall, I., Yamada, H., Bolas, M., & Debevec, P. (2007). Rendering for an interactive 360° light field display. ''ACM Transactions on Graphics (TOG)'', 26(3), 40-es.</ref>
====Integral Imaging Displays====
* '''Projector Arrays:''' Using multiple micro-projectors to beam images onto directional screens (e.g., lenticular sheets or [[anisotropic]] diffusers).
These use a [[microlens array]] placed over a high-resolution display panel (e.g., [[OLED]], [[LCD]]). Each microlens projects pixels underneath it into different directions, creating multiple views of the scene. Densely sampled views approximate a continuous light field, enabling [[autostereoscopic]] viewing.<ref name="Art1Martinez">Martinez-Corral, M., & Javidi, B. (2018). "Fundamentals of 3D imaging and displays: A tutorial on integral imaging, light-field, and plenoptic systems." Proceedings of the IEEE, 106(5), 891-908.</ref><ref name="Jones2007">Jones, A., McDowall, I., Yamada, H., Bolas, M., & Debevec, P. (2007). Rendering for an interactive 360° light field display. ''ACM Transactions on Graphics (TOG)'', 26(3), 40-es.</ref> This is effectively the inverse of a plenoptic camera.
* '''[[Holography|Holographic Optical Elements (HOEs)]]:''' Using diffractive optics to steer light rays appropriately.
 
* '''[[Volumetric display]]s:''' Creating a true 3D image in a volume of space, though often distinct from pure light field displays that reproduce rays intersecting a plane.
====Multi-Plane and Varifocal Displays====
These displays present images at multiple discrete focal planes, either simultaneously using stacked transparent displays or rapidly sequentially using [[varifocal lenses]] or switchable optics synchronized with [[eye tracking]].<ref name="Art1Mercier">Mercier, O., Sulai, Y., Mackenzie, K., Zannoli, M., Hillis, J., Nowrouzezahrai, D., & Lanman, D. (2017). "Fast gaze-contingent optimal decompositions for multifocal displays." ACM Transactions on Graphics, 36(6), 237.</ref> While not always full light field displays, they approximate the necessary focus cues. The Light Field Stereoscope is an example of a multi-layer approach.<ref name="Art1Huang"/><ref name="Art4MMCommSoc"/>
 
====Holographic Displays====
[[Holography|Holographic]] displays reconstruct the light wavefront itself using [[spatial light modulator]]s (SLMs) to control the phase or amplitude of light. These can, in theory, perfectly reproduce the light field of a scene, offering continuous focus cues.<ref name="Art1Li">Li, G., Lee, D., Jeong, Y., Cho, J., & Lee, B. (2016). "Holographic display for see-through augmented reality using mirror-lens holographic optical element." Optics Letters, 41(11), 2486-2489.</ref> Research includes using [[Holographic Optical Elements (HOEs)]] and [[metasurface]]s for compact designs, like Nvidia's Holographic Glasses prototype.<ref name="Art4NvidiaDev">[https://developer.nvidia.com/blog/prescription-holographic-vr-glasses-research/ Nvidia Developer Blog: Holographic Glasses Research]</ref>
 
====Compressive/Tensor Displays====
These use multiple layers of modulating panels (e.g., LCDs) with computational algorithms to sculpt the light passing through them, synthesizing a target light field with relatively thin hardware.<ref name="Wetzstein2011">Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2011). Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays. ''ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2011'', 30(4), 95:1-95:12.</ref><ref name="Art4MMCommSoc"/>
 
====Projector/Pinlight Arrays====
Systems using arrays of micro-projectors or scanned beams directed onto specialized screens (e.g., [[lenticular lens|lenticular sheets]]), or near-eye displays using arrays of "pinlights" (point sources imaged through microlenses or pinholes) can also generate light fields.<ref name="Art4MMCommSoc"/>


These display technologies aim to provide more realistic visual cues compared to traditional stereoscopic displays.
====Commercial Examples and Prototypes====
Several companies are developing light field or related display technologies:
* [[Looking Glass Factory]]: Produces tabletop holographic light field displays for glasses-free 3D.<ref name="Art1LookingGlass">Looking Glass Factory. (2021). "Technical Overview of Looking Glass Holographic Displays." Looking Glass Technical Documentation.</ref>
* [[Avegant]]: Developed [[retinal projection]] technology aiming for natural focus cues.<ref name="Art1Cheng">Cheng, D., Wang, Q., Liu, Y., & Wang, J. (2017). "Retinal projection type super multi-view 3D head-mounted display." Proceedings of SPIE Optical Engineering + Applications, 10335.</ref>
* [[Magic Leap]]: Their [[spatial computing]] headsets incorporate light field principles to blend virtual and real content.<ref name="Art1Kress">Kress, B. C., & Chatterjee, I. (2020). "Waveguide combiners for mixed reality headsets: a nanophotonics design perspective." Nanophotonics, 9(11), 3653-3667.</ref><ref name="Art2FXGuide">[https://www.fxguide.com/fxfeatured/light-fields-the-future-of-vr-ar-mr/ fxguide: Light Fields - The Future of VR-AR-MR]</ref>
* [[Leia Inc.]]: Creates light field displays for mobile devices.<ref name="Art1Fattal">Fattal, D., Peng, Z., Tran, T., Vo, S., Fiorentino, M., Brug, J., & Beausoleil, R. G. (2013). "A multi-directional backlight for a wide-angle, glasses-free three-dimensional display." Nature, 495(7441), 348-351.</ref>
* [[CREAL]]: Swiss startup developing near-eye light field displays specifically targeting the VAC issue in AR/VR.<ref name="Art2CrealRoad">[https://www.roadtovr.com/creal-light-field-display-new-immersion-ar/ Road to VR: Hands-on: CREAL's Light-field Display Brings a New Layer of Immersion to AR]</ref><ref name="Art2CrealSite">[https://creal.com/ CREAL: Light-field Display Technology]</ref>
* Light Field Lab: Developing large-scale holographic light field displays.


==Applications in VR and AR==
==Applications in VR and AR==
Light field technology holds immense promise for VR and AR by enabling more immersive and visually comfortable experiences:
Light field technology holds immense promise for VR and AR:


* '''Correct Parallax:''' Light fields inherently capture parallax information. Viewers using light field displays can move their heads slightly and see the scene perspective shift correctly, significantly enhancing realism and [[immersion]], particularly crucial for [[six degrees of freedom]] (6DoF) experiences.<ref name="Lanman2013">Lanman, D., & Luebke, D. (2013). Near-eye light field displays. ''ACM SIGGRAPH 2013 Talks'', 1-1.</ref>
* '''Addressing the [[Vergence-Accommodation Conflict]] (VAC):''' This is perhaps the most significant benefit. By providing correct focus cues matching the [[vergence]] depth, light field displays can reduce eye strain, nausea, and visual fatigue associated with prolonged use of conventional stereoscopic HMDs, leading to more comfortable and immersive experiences.<ref name="Konrad2017"/><ref name="Art2CrealRoad"/>
* '''View-dependent Effects:''' Complex interactions of light, such as [[reflection]]s and [[refraction]]s, change based on viewpoint. Light fields capture these effects, allowing them to be reproduced accurately in VR/AR headsets.
* '''Correct Parallax and 6DoF:''' Light fields inherently support correct parallax shifts as the user moves their head within the [[eye-box]], crucial for realistic [[six degrees of freedom]] (6DoF) experiences in VR and stable anchoring of virtual objects in AR.<ref name="Lanman2013"/> Google's "Welcome to Light Fields" VR app demonstrated this effect.<ref name="Art2GoogleBlog">[https://blog.google/products/google-ar-vr/experimenting-light-fields/ Google Blog: Experimenting with Light Fields]</ref>
* '''Post-Capture Refocusing and DoF Control:''' Light field recordings allow users or applications to change focus or DoF after the fact, which could be used for cinematic effects, accessibility features, or gaze-tracked rendering.
* '''Enhanced Visual Fidelity and View-Dependent Effects:''' Light fields capture and reproduce complex light interactions like specular [[highlight]]s, transparency, reflections, and refractions more accurately than traditional rendering, enhancing realism.<ref name="Art1Mildenhall"/>
* '''Addressing the [[Vergence-Accommodation Conflict]] (VAC):''' Conventional stereoscopic displays present conflicting depth cues: the eyes converge at the depth of the virtual object, but accommodate (focus) at the fixed distance of the display screen. This mismatch can cause eye strain and nausea. Light field displays aim to present light rays that appear to originate from the correct depths, allowing the eye to focus naturally and potentially mitigating the VAC.<ref name="Konrad2017">Konrad, R., Cooper, E. A., Wetzstein, G., & Banks, M. S. (2017). Accommodation and vergence responses to near-eye light field displays. ''Journal of Vision'', 17(10), 987-987.</ref>
* '''Realistic Capture for VR/AR Content:''' Light field cameras capture real-world scenes with richer information than 360° video or basic [[photogrammetry]], preserving subtle lighting and allowing more natural exploration in VR. Systems like [[Google]]'s light field capture rigs and [[Lytro]] Immerge were developed for this.<ref name="Art2GoogleBlog"/>
* '''Realistic Capture for VR/AR Content:''' Light field cameras can capture real-world scenes that can be explored more naturally in VR/AR than traditional 360° video or [[photogrammetry]] models, preserving subtle lighting effects.
* '''[[Light Field Passthrough]] for Mixed Reality:''' An emerging technique for AR/[[Mixed Reality|MR]] headsets where specialized cameras capture the light field of the real world. This allows rendering the outside view with correct depth and perspective for the user's eyes, enabling seamless blending of virtual objects with reality and minimizing reprojection errors or distortions seen in traditional video passthrough. Meta's Flamera prototype is a notable example.<ref name="Art2TeknoAsian">[https://teknoasian.com/light-field-passthrough-the-bridge-between-reality-and-virtual-worlds/ Tekno Asian: Light Field Passthrough: The Bridge Between Reality and Virtual Worlds]</ref><ref name="Art4Spectrum">[https://spectrum.ieee.org/meta-flamera IEEE Spectrum: Meta Builds AR Headset With Unrivaled Passthrough]</ref><ref name="Art4DisplayDaily">[https://www.displaydaily.com/article/display-daily/metas-perspective-correct-passthrough-mr-display Display Daily: Meta’s Perspective-Correct Passthrough MR Display]</ref>
* '''Enhanced [[Depth Estimation]]:''' The angular information in light fields provides strong cues for calculating accurate depth maps of captured scenes.
* '''[[Telepresence]] and Remote Collaboration:''' Realistic capture and display of participants using light fields can significantly enhance the sense of presence in virtual meetings and remote collaboration systems, enabling more natural eye contact and spatial interaction.<ref name="Art1Orts">Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W., Kowdle, A., Degtyarev, Y., Kim, D., Davidson, P. L., Khamis, S., Dou, M., Tankovich, V., Loop, C., Cai, Q., Chou, P. A., Mennicken, S., Valentin, J., Pradeep, V., Wang, S., Kang, S. B., Kohli, P., Lutchyn, Y., Keskin, C., & Izadi, S. (2016). "Holoportation: Virtual 3D teleportation in real-time." Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 741-754.</ref>
* '''Post-Capture Refocus and DoF Control:''' While primarily a photographic benefit, this capability could be used in VR/AR for cinematic effects, accessibility features, or interactive storytelling.


==Advantages==
==Advantages==
* Provides highly realistic views with correct parallax and view-dependent effects.
* Provides highly realistic views with correct parallax and view-dependent effects.
* Enables post-capture refocusing and depth of field adjustments.
* Enables post-capture refocusing and depth of field adjustments (primarily capture advantage).
* Potential to significantly reduce or eliminate the vergence-accommodation conflict in HMDs.
* Potential to significantly reduce or eliminate the vergence-accommodation conflict in HMDs, increasing comfort.
* Captures rich scene information useful for various computational photography tasks.
* Captures rich scene information useful for various computational photography and computer vision tasks (e.g., depth estimation).
* Enables more seamless integration of virtual elements in AR/MR via techniques like light field passthrough.


==Challenges and Limitations==
==Challenges and Limitations==
* '''Data Size:''' 4D light fields represent significantly more data than conventional 2D images or even stereo pairs, posing challenges for capture, storage, transmission, and processing.
* '''Data Size and Management:''' 4D light fields represent significantly more data than conventional images, posing challenges for capture, storage (often terabytes for high quality), transmission (streaming), and processing.<ref name="Art1Alain">Alain, M., Smolic, A., & Guillemot, C. (2019). "Light field compression: A review." IEEE Journal of Selected Topics in Signal Processing, 13(3), 454-463.</ref>
* '''Capture Hardware Complexity:''' Building high-resolution light field cameras or camera arrays is complex and often expensive. Achieving wide [[field of view]] (FoV) and high angular resolution simultaneously is difficult.
* '''Computational Complexity:''' Processing and rendering light fields, especially in real-time for high-resolution VR/AR, requires substantial computational power. Optimization and [[machine learning]] approaches are active research areas.<ref name="Art1Wang">Wang, T. C., Efros, A. A., & Ramamoorthi, R. (2021). "Neural rendering and neural light transport for mixed reality." IEEE Transactions on Visualization and Computer Graphics, 27(5), 2657-2671.</ref>
* '''Display Technology Immaturity:''' High-resolution, high-brightness, wide FoV light field displays suitable for consumer VR/AR are still largely in the research and development phase. Current prototypes often face trade-offs between resolution, brightness, FoV, and computational cost.
* '''Capture Hardware Complexity and Cost:''' High-quality light field capture systems (plenoptic cameras, large camera arrays) remain complex, expensive, and often limited to controlled environments.<ref name="Art1Overbeck">Overbeck, R. S., Erickson, D., Evangelakos, D., Pharr, M., & Debevec, P. (2018). "A system for acquiring, processing, and rendering panoramic light field stills for virtual reality." ACM Transactions on Graphics, 37(6), 197.</ref>
* '''Computational Cost:''' Rendering light fields, especially in real-time for VR/AR, requires significant computational power. Efficient compression and rendering algorithms are crucial.
* '''Display Technology Immaturity and Trade-offs:''' High-performance light field displays suitable for consumer VR/AR HMDs (high resolution, high brightness, wide [[field of view]] (FoV), large eye-box, low latency, compact form factor) are still largely under development. Current technologies often involve trade-offs, e.g., between spatial and angular resolution.<ref name="Art1Wetzstein">Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2012). "Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting." ACM Transactions on Graphics, 31(4), 80.</ref><ref name="Art4MMCommSoc"/>
* '''Limited Angular Resolution:''' Current practical systems often have limited angular resolution, which can constrain the range of viewpoint movement and the effectiveness in resolving VAC.
* '''Limited Angular Resolution:''' Practical systems often have limited angular resolution, which can restrict the range of parallax and the effectiveness in fully resolving VAC.
* '''Eye-Box Size:''' Some display approaches (especially holographic and integral imaging) can have a limited viewing zone (eye-box) where the effect is perceived correctly, requiring precise alignment or [[eye tracking]] compensation.<ref name="Art4MMCommSoc"/>
 
==Future Directions==
Research and development continue to advance light field technology:
 
* '''Neural Radiance Fields (NeRF) and Neural Rendering:''' These [[machine learning]] techniques are rapidly evolving, offering efficient ways to represent and render complex scenes with view-dependent effects, potentially revolutionizing light field capture and synthesis for VR/AR.<ref name="Art1Mildenhall"/>
* '''Eye-Tracked Foveated Light Fields:''' Combining [[eye tracking]] with light field rendering/display allows concentrating detail and computational resources where the user is looking ([[foveated rendering]]), making real-time performance more feasible.<ref name="Art1Kaplanyan">Kaplanyan, A. S., Sochenov, A., Leimkühler, T., Okunev, M., Goodall, T., & Rufo, G. (2019). "DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos." ACM Transactions on Graphics, 38(6), 212.</ref><ref name="Art4NvidiaResearch">[https://research.nvidia.com/publication/2017-11_Foveated-Light-field-Rendering Nvidia Research: Foveated Light-field Rendering]</ref>
* '''Compact Light Field Optics:''' Development of [[metalenses]], [[diffractive optics]], novel [[waveguide]] designs, and HOEs aims to create thinner, lighter, and more efficient optics for near-eye light field displays suitable for glasses-like AR/VR devices.<ref name="Art1WangOptics">Wang, N., Hua, H., & Viegas, D. (2021). "Compact optical see-through head-mounted display with varifocal liquid membrane lens." Digital Holography and Three-Dimensional Imaging 2021, OSA Technical Digest, DM3B.3.</ref><ref name="Art4NvidiaDev"/>
* '''Light Field Video Streaming:''' Advances in compression and network bandwidth may enable real-time streaming of light field video for immersive communication, entertainment, and training.<ref name="Art1Gutierrez">Gutiérrez-Navarro, D., & Pérez-Daniel, K. R. (2022). "Light field video streaming: A review." IEEE Access, 10, 12345-12367.</ref>
* '''Integration with Haptics and Spatial Audio:''' Combining high-fidelity light field visuals with [[haptic feedback]] and [[spatial audio]] promises truly immersive multisensory experiences.<ref name="Art1Martins">Martins, R., Terziman, L., Paljic, A., & Pettré, J. (2021). "Integrating vision, haptics and proprioception into a unified 3D perception framework for immersive virtual reality." IEEE Transactions on Haptics, 14(2), 401-410.</ref>


==See Also==
==See Also==
Line 82: Line 121:
* [[Light field camera]]
* [[Light field camera]]
* [[Light field display]]
* [[Light field display]]
* [[Light Field Passthrough]]
* [[Computational photography]]
* [[Computational photography]]
* [[Virtual Reality]] (VR)
* [[Virtual Reality]] (VR)
* [[Augmented Reality]] (AR)
* [[Augmented Reality]] (AR)
* [[Mixed Reality]] (MR)
* [[Six degrees of freedom]] (6DoF)
* [[Six degrees of freedom]] (6DoF)
* [[Parallax]]
* [[Parallax]]
* [[Vergence-accommodation conflict]]
* [[Vergence-accommodation conflict]] (VAC)
* [[Accommodation (eye)]]
* [[Vergence]]
* [[Holography]]
* [[Holography]]
* [[Integral photography]]
* [[Neural Radiance Fields]] (NeRF)
* [[Computer graphics]]
* [[Computer graphics]]
* [[Computer vision]]
* [[Computer vision]]
* [[Head-mounted display]] (HMD)
* [[Eye tracking]]
* [[Foveated rendering]]


==References==
==References==
<references />
<references />

Revision as of 02:08, 27 April 2025

A light field (also spelled lightfield) is a fundamental concept in optics and computer graphics that describes the amount of light traveling in every direction through every point in space.[1][2] Essentially, it's a vector function that represents the radiance of light rays at any position and direction within a given volume or area. Understanding and utilizing light fields is crucial for advancing virtual reality (VR) and augmented reality (AR) technologies, as it allows for the capture and reproduction of visual scenes with unprecedented realism, including effects like parallax, reflections, refractions, and refocusing after capture, while also aiming to solve critical issues like the vergence-accommodation conflict.[3][4]

History

The concept of measuring light rays has early roots. Michael Faraday first speculated in 1846 in his lecture "Thoughts on Ray Vibrations" that light should be understood as a field, similar to the magnetic field he had studied.[5] The term "light field" (svetovoe pole in Russian) was more formally defined by Andrey Gershun in a classic 1936 paper on the radiometric properties of light in three-dimensional space.[6][7]

In the context of computer vision and graphics, the concept was further developed with the introduction of the 7D plenoptic function by Adelson and Bergen in 1991.[8] This function describes all possible light rays, parameterized by 3D position (x, y, z), 2D direction (θ, φ), wavelength (λ), and time (t).

Practical computational approaches often reduce the dimensionality. Two seminal papers in 1996, "Light Field Rendering" by Levoy and Hanrahan at Stanford University[1] and "The Lumigraph" by Gortler et al.[2], independently proposed using a 4D subset of the plenoptic function for capturing and rendering complex scenes without detailed 3D models.[9] They introduced the highly influential two-plane parameterization (2PP), simplifying the representation by defining rays based on their intersection points with two parallel planes. This 4D light field representation forms the basis for most modern light field capture and display technologies. The introduction of light field cameras, inspired by Gabriel Lippmann’s 1908 concept of integral photography, further advanced the field.[10]

Theory and Representation

The core idea behind the light field is to capture not just the intensity of light arriving at a point (like a conventional camera), but also the direction from which that light is arriving.

The Plenoptic Function

The most complete representation is the 7D plenoptic function, P(x, y, z, θ, φ, λ, t), describing the radiance of light at any 3D point (x,y,z), in any direction (θ, φ), for any wavelength (λ), at any time (t).[8] For many applications, this is overly complex and contains redundant information (e.g., light doesn't typically change along a straight ray in free space—radiance invariance—unless wavelength or time are critical).[7]

Simplified Light Fields

For static scenes under constant illumination, the time (t) and wavelength (λ, often simplified to RGB channels) dependencies can often be dropped. Furthermore, due to the constancy of radiance along a ray in free space, the 3D spatial component can be reduced.

  • 5D Light Field: Often represented as L = L(x, y, z, θ, φ).[11]
  • 4D Light Field: The most common simplification, often called the photic field or lumigraph in regions free of occluders.[7] It captures radiance along rays without redundant data.[1][2]

Two-Plane Parameterization (2PP)

This popular 4D parameterization defines a light ray by its intersection points with two arbitrary planes, often denoted as the (u, v) plane and the (s, t) plane. A ray is thus uniquely identified by the coordinates L(u, v, s, t).[1] This representation is convenient because:

  • It relates well to how light field cameras with microlens arrays capture data.
  • It simplifies rendering algorithms, which often involve resampling this 4D function.

Other 4D parameterizations exist, such as using one point and two angles, or using spherical coordinates.

Light Field Capture

Capturing a light field involves sampling the intensity and direction of light rays within a scene. Several methods exist:

  • Light Field Cameras (Plenoptic Cameras): These are common devices that typically insert a microlens array between the main lens and the image sensor.[12][3] Each microlens samples the light arriving from the main lens from slightly different perspectives, directing these samples onto different pixels on the sensor below it. The sensor thus records not only spatial information but also angular information.[10] Consumer examples included cameras from Lytro (now defunct) and Raytrix.[3]
  • Camera Arrays: A synchronized array of conventional cameras can capture a light field by recording the scene from slightly different viewpoints simultaneously.[13] This approach often yields higher spatial resolution but requires careful calibration and synchronization.[14]
  • Scanning/Gantry Setups: A single camera moved precisely to multiple positions can sequentially capture the views needed to sample the light field. This is suitable for static scenes but slower.[2]
  • Coded Aperture Photography: Placing a patterned mask (coded aperture) near the sensor or aperture can modulate incoming light in a way that allows directional information to be computationally recovered.[15]
  • Synthetic Light Fields: In computer-generated imagery (CGI), light fields can be generated synthetically by rendering a scene from multiple viewpoints. This is particularly relevant for VR and AR where scenes might be entirely or partially virtual.[16]
  • Computational Photography Techniques: Various other methods combining optics and computation are continuously being developed.

Light Field Processing

Raw light field data requires significant processing:

  • Calibration and Rectification: Raw data needs calibration to correct for lens distortions, sensor noise, and camera misalignments, especially in camera arrays or plenoptic systems. Geometric calibration ensures accurate reconstruction.[17]
  • Interpolation and View Synthesis: A key advantage is generating novel viewpoints not explicitly captured. This involves interpolating the 4D light field data to estimate the scene's appearance from arbitrary positions and angles.[18]
  • Depth Estimation: The angular variation of light rays encodes depth information. Various algorithms can extract depth maps, valuable for effects like synthetic depth of field and for AR interactions.[19]
  • Compression: Light field datasets are massive. Efficient compression is vital for storage and transmission, especially for mobile VR/AR and streaming. Techniques often adapt existing video codecs (like VP9) or use specialized approaches.[20][21] Standards bodies like JPEG Pleno and MPEG Immersive Video are developing formats for light field data.[22]

Light Field Rendering and Display

Once captured or generated, light fields enable novel rendering and display possibilities.

Rendering

Rendering involves sampling the 4D light field data to synthesize a 2D image for a virtual camera. For a desired camera pose, the algorithm determines which rays intersect the virtual sensor and integrates their values.[1] This enables:

  • Refocusing: Synthetically changing the focal plane after capture.[3]
  • Depth of Field Control: Computationally adjusting the aperture size.
  • View Synthesis: Generating views from slightly different positions, enabling parallax effects and six degrees of freedom (6DoF) movement.[4]
  • Neural Radiance Fields (NeRF) and Neural Rendering: Modern approaches like NeRF represent the scene as a continuous function learned by a neural network, allowing high-quality view synthesis from sparse inputs. These methods effectively learn a compressed light field representation and are highly relevant for VR/AR rendering.[23][24]

Light Field Displays

Light field displays aim to reproduce the directional aspect of light rays, allowing the viewer's eyes to accommodate (focus) naturally at different depths, potentially resolving the vergence-accommodation conflict inherent in conventional stereoscopic displays.[25][26] This provides more natural depth cues and improves visual comfort. Key approaches include:

Integral Imaging Displays

These use a microlens array placed over a high-resolution display panel (e.g., OLED, LCD). Each microlens projects pixels underneath it into different directions, creating multiple views of the scene. Densely sampled views approximate a continuous light field, enabling autostereoscopic viewing.[27][28] This is effectively the inverse of a plenoptic camera.

Multi-Plane and Varifocal Displays

These displays present images at multiple discrete focal planes, either simultaneously using stacked transparent displays or rapidly sequentially using varifocal lenses or switchable optics synchronized with eye tracking.[29] While not always full light field displays, they approximate the necessary focus cues. The Light Field Stereoscope is an example of a multi-layer approach.[25][22]

Holographic Displays

Holographic displays reconstruct the light wavefront itself using spatial light modulators (SLMs) to control the phase or amplitude of light. These can, in theory, perfectly reproduce the light field of a scene, offering continuous focus cues.[30] Research includes using Holographic Optical Elements (HOEs) and metasurfaces for compact designs, like Nvidia's Holographic Glasses prototype.[31]

Compressive/Tensor Displays

These use multiple layers of modulating panels (e.g., LCDs) with computational algorithms to sculpt the light passing through them, synthesizing a target light field with relatively thin hardware.[32][22]

Projector/Pinlight Arrays

Systems using arrays of micro-projectors or scanned beams directed onto specialized screens (e.g., lenticular sheets), or near-eye displays using arrays of "pinlights" (point sources imaged through microlenses or pinholes) can also generate light fields.[22]

Commercial Examples and Prototypes

Several companies are developing light field or related display technologies:

Applications in VR and AR

Light field technology holds immense promise for VR and AR:

  • Addressing the Vergence-Accommodation Conflict (VAC): This is perhaps the most significant benefit. By providing correct focus cues matching the vergence depth, light field displays can reduce eye strain, nausea, and visual fatigue associated with prolonged use of conventional stereoscopic HMDs, leading to more comfortable and immersive experiences.[26][38]
  • Correct Parallax and 6DoF: Light fields inherently support correct parallax shifts as the user moves their head within the eye-box, crucial for realistic six degrees of freedom (6DoF) experiences in VR and stable anchoring of virtual objects in AR.[4] Google's "Welcome to Light Fields" VR app demonstrated this effect.[40]
  • Enhanced Visual Fidelity and View-Dependent Effects: Light fields capture and reproduce complex light interactions like specular highlights, transparency, reflections, and refractions more accurately than traditional rendering, enhancing realism.[23]
  • Realistic Capture for VR/AR Content: Light field cameras capture real-world scenes with richer information than 360° video or basic photogrammetry, preserving subtle lighting and allowing more natural exploration in VR. Systems like Google's light field capture rigs and Lytro Immerge were developed for this.[40]
  • Light Field Passthrough for Mixed Reality: An emerging technique for AR/MR headsets where specialized cameras capture the light field of the real world. This allows rendering the outside view with correct depth and perspective for the user's eyes, enabling seamless blending of virtual objects with reality and minimizing reprojection errors or distortions seen in traditional video passthrough. Meta's Flamera prototype is a notable example.[41][42][43]
  • Telepresence and Remote Collaboration: Realistic capture and display of participants using light fields can significantly enhance the sense of presence in virtual meetings and remote collaboration systems, enabling more natural eye contact and spatial interaction.[44]
  • Post-Capture Refocus and DoF Control: While primarily a photographic benefit, this capability could be used in VR/AR for cinematic effects, accessibility features, or interactive storytelling.

Advantages

  • Provides highly realistic views with correct parallax and view-dependent effects.
  • Enables post-capture refocusing and depth of field adjustments (primarily capture advantage).
  • Potential to significantly reduce or eliminate the vergence-accommodation conflict in HMDs, increasing comfort.
  • Captures rich scene information useful for various computational photography and computer vision tasks (e.g., depth estimation).
  • Enables more seamless integration of virtual elements in AR/MR via techniques like light field passthrough.

Challenges and Limitations

  • Data Size and Management: 4D light fields represent significantly more data than conventional images, posing challenges for capture, storage (often terabytes for high quality), transmission (streaming), and processing.[45]
  • Computational Complexity: Processing and rendering light fields, especially in real-time for high-resolution VR/AR, requires substantial computational power. Optimization and machine learning approaches are active research areas.[46]
  • Capture Hardware Complexity and Cost: High-quality light field capture systems (plenoptic cameras, large camera arrays) remain complex, expensive, and often limited to controlled environments.[47]
  • Display Technology Immaturity and Trade-offs: High-performance light field displays suitable for consumer VR/AR HMDs (high resolution, high brightness, wide field of view (FoV), large eye-box, low latency, compact form factor) are still largely under development. Current technologies often involve trade-offs, e.g., between spatial and angular resolution.[48][22]
  • Limited Angular Resolution: Practical systems often have limited angular resolution, which can restrict the range of parallax and the effectiveness in fully resolving VAC.
  • Eye-Box Size: Some display approaches (especially holographic and integral imaging) can have a limited viewing zone (eye-box) where the effect is perceived correctly, requiring precise alignment or eye tracking compensation.[22]

Future Directions

Research and development continue to advance light field technology:

  • Neural Radiance Fields (NeRF) and Neural Rendering: These machine learning techniques are rapidly evolving, offering efficient ways to represent and render complex scenes with view-dependent effects, potentially revolutionizing light field capture and synthesis for VR/AR.[23]
  • Eye-Tracked Foveated Light Fields: Combining eye tracking with light field rendering/display allows concentrating detail and computational resources where the user is looking (foveated rendering), making real-time performance more feasible.[49][50]
  • Compact Light Field Optics: Development of metalenses, diffractive optics, novel waveguide designs, and HOEs aims to create thinner, lighter, and more efficient optics for near-eye light field displays suitable for glasses-like AR/VR devices.[51][31]
  • Light Field Video Streaming: Advances in compression and network bandwidth may enable real-time streaming of light field video for immersive communication, entertainment, and training.[52]
  • Integration with Haptics and Spatial Audio: Combining high-fidelity light field visuals with haptic feedback and spatial audio promises truly immersive multisensory experiences.[53]

See Also

References

  1. ↑ 1.0 1.1 1.2 1.3 1.4 Levoy, M., & Hanrahan, P. (1996). Light field rendering. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96, 31–42.
  2. ↑ 2.0 2.1 2.2 2.3 Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996). The Lumigraph. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96, 43–54.
  3. ↑ 3.0 3.1 3.2 3.3 Ng, R. (2005). Digital Light Field Photography. Ph.D. Thesis, Stanford University.
  4. ↑ 4.0 4.1 4.2 Lanman, D., & Luebke, D. (2013). Near-eye light field displays. ACM SIGGRAPH 2013 Talks, 1-1.
  5. ↑ Faraday, M. (1846). Thoughts on Ray Vibrations. Philosophical Magazine, S.3, Vol. 28, No. 188.
  6. ↑ Gershun, A. (1939). The Light Field. Journal of Mathematics and Physics, 18(1-4), 51–151. (English translation of 1936 Russian paper).
  7. ↑ 7.0 7.1 7.2 Wikipedia: Light field
  8. ↑ 8.0 8.1 Adelson, E. H., & Bergen, J. R. (1991). The plenoptic function and the elements of early vision. In Computational Models of Visual Processing (pp. 3-20). MIT Press.
  9. ↑ Stanford University: Light fields and computational photography
  10. ↑ 10.0 10.1 Wikipedia: Light field camera
  11. ↑ Faraday, M. (1846). "Experimental Researches in Electricity." Philosophical Transactions of the Royal Society of London, 136, 1-20.
  12. ↑ Adelson, E. H., & Wang, J. Y. A. (1992). Single lens stereo with a plenoptic camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 99-106.
  13. ↑ Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., ... & Levoy, M. (2005). High performance imaging using large camera arrays. ACM Transactions on Graphics (TOG), 24(3), 765-776.
  14. ↑ Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., Adams, A., Horowitz, M., & Levoy, M. (2005). "High performance imaging using large camera arrays." ACM Transactions on Graphics, 24(3), 765-776.
  15. ↑ Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007). Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Transactions on Graphics (TOG), 26(3), 69-es.
  16. ↑ Moon, B., McDermott, S., Golas, K., Cao, Y., & Wong, T.-T. (2016). "Efficient rendering of virtual objects for mixed reality." Computers & Graphics, 57, 93-101.
  17. ↑ Bok, Y., Jeon, H. G., & Kweon, I. S. (2017). "Geometric calibration of micro-lens-based light field cameras using line features." IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(2), 287-300.
  18. ↑ Kalantari, N. K., Wang, T. C., & Ramamoorthi, R. (2016). "Learning-based view synthesis for light field cameras." ACM Transactions on Graphics, 35(6), 193.
  19. ↑ Tao, M. W., Hadap, S., Malik, J., & Ramamoorthi, R. (2013). "Depth from combining defocus and correspondence using light-field cameras." Proceedings of the IEEE International Conference on Computer Vision, 673-680.
  20. ↑ Viola, I., Rerabek, M., & Ebrahimi, T. (2017). "Comparison and evaluation of light field image coding approaches." IEEE Journal of Selected Topics in Signal Processing, 11(7), 1092-1106.
  21. ↑ Augmented Perception: Welcome to Light Fields
  22. ↑ 22.0 22.1 22.2 22.3 22.4 22.5 IEEE ComSoc MMTC Review Letter, Vol. 8, No. 2, Nov 2017
  23. ↑ 23.0 23.1 23.2 Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). "NeRF: Representing scenes as neural radiance fields for view synthesis." European Conference on Computer Vision, 405-421.
  24. ↑ VR-NeRF Project Page
  25. ↑ 25.0 25.1 Huang, F. C., Chen, K., & Wetzstein, G. (2015). "The light field stereoscope: Immersive computer graphics via factored near-eye light field displays with focus cues." ACM Transactions on Graphics, 34(4), 60.
  26. ↑ 26.0 26.1 Konrad, R., Cooper, E. A., Wetzstein, G., & Banks, M. S. (2017). Accommodation and vergence responses to near-eye light field displays. Journal of Vision, 17(10), 987-987.
  27. ↑ Martinez-Corral, M., & Javidi, B. (2018). "Fundamentals of 3D imaging and displays: A tutorial on integral imaging, light-field, and plenoptic systems." Proceedings of the IEEE, 106(5), 891-908.
  28. ↑ Jones, A., McDowall, I., Yamada, H., Bolas, M., & Debevec, P. (2007). Rendering for an interactive 360° light field display. ACM Transactions on Graphics (TOG), 26(3), 40-es.
  29. ↑ Mercier, O., Sulai, Y., Mackenzie, K., Zannoli, M., Hillis, J., Nowrouzezahrai, D., & Lanman, D. (2017). "Fast gaze-contingent optimal decompositions for multifocal displays." ACM Transactions on Graphics, 36(6), 237.
  30. ↑ Li, G., Lee, D., Jeong, Y., Cho, J., & Lee, B. (2016). "Holographic display for see-through augmented reality using mirror-lens holographic optical element." Optics Letters, 41(11), 2486-2489.
  31. ↑ 31.0 31.1 Nvidia Developer Blog: Holographic Glasses Research
  32. ↑ Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2011). Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2011, 30(4), 95:1-95:12.
  33. ↑ Looking Glass Factory. (2021). "Technical Overview of Looking Glass Holographic Displays." Looking Glass Technical Documentation.
  34. ↑ Cheng, D., Wang, Q., Liu, Y., & Wang, J. (2017). "Retinal projection type super multi-view 3D head-mounted display." Proceedings of SPIE Optical Engineering + Applications, 10335.
  35. ↑ Kress, B. C., & Chatterjee, I. (2020). "Waveguide combiners for mixed reality headsets: a nanophotonics design perspective." Nanophotonics, 9(11), 3653-3667.
  36. ↑ fxguide: Light Fields - The Future of VR-AR-MR
  37. ↑ Fattal, D., Peng, Z., Tran, T., Vo, S., Fiorentino, M., Brug, J., & Beausoleil, R. G. (2013). "A multi-directional backlight for a wide-angle, glasses-free three-dimensional display." Nature, 495(7441), 348-351.
  38. ↑ 38.0 38.1 Road to VR: Hands-on: CREAL's Light-field Display Brings a New Layer of Immersion to AR
  39. ↑ CREAL: Light-field Display Technology
  40. ↑ 40.0 40.1 Google Blog: Experimenting with Light Fields
  41. ↑ Tekno Asian: Light Field Passthrough: The Bridge Between Reality and Virtual Worlds
  42. ↑ IEEE Spectrum: Meta Builds AR Headset With Unrivaled Passthrough
  43. ↑ Display Daily: Meta’s Perspective-Correct Passthrough MR Display
  44. ↑ Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W., Kowdle, A., Degtyarev, Y., Kim, D., Davidson, P. L., Khamis, S., Dou, M., Tankovich, V., Loop, C., Cai, Q., Chou, P. A., Mennicken, S., Valentin, J., Pradeep, V., Wang, S., Kang, S. B., Kohli, P., Lutchyn, Y., Keskin, C., & Izadi, S. (2016). "Holoportation: Virtual 3D teleportation in real-time." Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 741-754.
  45. ↑ Alain, M., Smolic, A., & Guillemot, C. (2019). "Light field compression: A review." IEEE Journal of Selected Topics in Signal Processing, 13(3), 454-463.
  46. ↑ Wang, T. C., Efros, A. A., & Ramamoorthi, R. (2021). "Neural rendering and neural light transport for mixed reality." IEEE Transactions on Visualization and Computer Graphics, 27(5), 2657-2671.
  47. ↑ Overbeck, R. S., Erickson, D., Evangelakos, D., Pharr, M., & Debevec, P. (2018). "A system for acquiring, processing, and rendering panoramic light field stills for virtual reality." ACM Transactions on Graphics, 37(6), 197.
  48. ↑ Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2012). "Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting." ACM Transactions on Graphics, 31(4), 80.
  49. ↑ Kaplanyan, A. S., Sochenov, A., Leimkühler, T., Okunev, M., Goodall, T., & Rufo, G. (2019). "DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos." ACM Transactions on Graphics, 38(6), 212.
  50. ↑ Nvidia Research: Foveated Light-field Rendering
  51. ↑ Wang, N., Hua, H., & Viegas, D. (2021). "Compact optical see-through head-mounted display with varifocal liquid membrane lens." Digital Holography and Three-Dimensional Imaging 2021, OSA Technical Digest, DM3B.3.
  52. ↑ Gutiérrez-Navarro, D., & Pérez-Daniel, K. R. (2022). "Light field video streaming: A review." IEEE Access, 10, 12345-12367.
  53. ↑ Martins, R., Terziman, L., Paljic, A., & Pettré, J. (2021). "Integrating vision, haptics and proprioception into a unified 3D perception framework for immersive virtual reality." IEEE Transactions on Haptics, 14(2), 401-410.
Retrieved from "https://vrarwiki.com/index.php?title=Light_field&oldid=34582"