Jump to content

Light field

From VR & AR Wiki
See also: Terms and Technical Terms

A light field (also spelled lightfield) is a fundamental concept in optics and computer graphics that describes the amount of light traveling in every direction through every point in space.[1][2] Essentially, it's a vector function that represents the radiance of light rays at any position and direction within a given volume or area. Understanding and utilizing light fields is crucial for advancing virtual reality (VR) and augmented reality (AR) technologies, as it allows for the capture and reproduction of visual scenes with unprecedented realism, including effects like parallax, reflections, refractions, and refocusing after capture, while also aiming to solve critical issues like the vergence-accommodation conflict.[3][4]

History

The concept of measuring light rays has early roots. Michael Faraday first speculated in 1846 in his lecture "Thoughts on Ray Vibrations" that light should be understood as a field, similar to the magnetic field he had studied.[5] The term "light field" (svetovoe pole in Russian) was more formally defined by Andrey Gershun in a classic 1936 paper on the radiometric properties of light in three-dimensional space.[6][7]

In the context of computer vision and graphics, the concept was further developed with the introduction of the 7D plenoptic function by Adelson and Bergen in 1991.[8] This function describes all possible light rays, parameterized by 3D position (x, y, z), 2D direction (θ, φ), wavelength (λ), and time (t).

Practical computational approaches often reduce the dimensionality. Two seminal papers in 1996, "Light Field Rendering" by Levoy and Hanrahan at Stanford University[1] and "The Lumigraph" by Gortler et al.[2], independently proposed using a 4D subset of the plenoptic function for capturing and rendering complex scenes without detailed 3D models.[9] They introduced the highly influential two-plane parameterization (2PP), simplifying the representation by defining rays based on their intersection points with two parallel planes. This 4D light field representation forms the basis for most modern light field capture and display technologies. The introduction of light field cameras, inspired by Gabriel Lippmann’s 1908 concept of integral photography, further advanced the field.[10]

Theory and Representation

The core idea behind the light field is to capture not just the intensity of light arriving at a point (like a conventional camera), but also the direction from which that light is arriving.

The Plenoptic Function

The most complete representation is the 7D plenoptic function, P(x, y, z, θ, φ, λ, t), describing the radiance of light at any 3D point (x,y,z), in any direction (θ, φ), for any wavelength (λ), at any time (t).[8] For many applications, this is overly complex and contains redundant information (for example light doesn't typically change along a straight ray in free space—radiance invariance—unless wavelength or time are critical).[7]

Simplified Light Fields

For static scenes under constant illumination, the time (t) and wavelength (λ, often simplified to RGB channels) dependencies can often be dropped. Furthermore, due to the constancy of radiance along a ray in free space, the 3D spatial component can be reduced.

  • 5D Light Field: Often represented as L = L(x, y, z, θ, φ).
  • 4D Light Field: The most common simplification, often called the photic field or lumigraph in regions free of occluders.[7] It captures radiance along rays without redundant data.[1][2]

Two-Plane Parameterization (2PP)

This popular 4D parameterization defines a light ray by its intersection points with two arbitrary planes, often denoted as the (u, v) plane and the (s, t) plane. A ray is thus uniquely identified by the coordinates L(u, v, s, t).[1] This representation is convenient because:

  • It relates well to how light field cameras with microlens arrays capture data.
  • It simplifies rendering algorithms, which often involve resampling this 4D function.

Other 4D parameterizations exist, such as using one point and two angles, or using spherical coordinates.

Light Field Capture

Capturing a light field involves sampling the intensity and direction of light rays within a scene. Several methods exist:

  • Light Field Cameras (Plenoptic Cameras): These are common devices that typically insert a microlens array between the main lens and the image sensor.[11][3] Each microlens samples the light arriving from the main lens from slightly different perspectives, directing these samples onto different pixels on the sensor below it. The sensor thus records not only spatial information but also angular information.[10] Consumer examples included cameras from Lytro (now defunct) and Raytrix.[3]
  • Camera Arrays: A synchronized array of conventional cameras can capture a light field by recording the scene from slightly different viewpoints simultaneously.[12] This approach often yields higher spatial resolution but requires careful calibration and synchronization.[13]
  • Scanning/Gantry Setups: A single camera moved precisely to multiple positions can sequentially capture the views needed to sample the light field. This is suitable for static scenes but slower.[2]
  • Coded Aperture Photography: Placing a patterned mask (coded aperture) near the sensor or aperture can modulate incoming light in a way that allows directional information to be computationally recovered.[14]
  • Synthetic Light Fields: In computer-generated imagery (CGI), light fields can be generated synthetically by rendering a scene from multiple viewpoints. This is particularly relevant for VR and AR where scenes might be entirely or partially virtual.[15]
  • Computational Photography Techniques: Various other methods combining optics and computation are continuously being developed.

Light Field Processing

Raw light field data requires significant processing:

  • Calibration and Rectification: Raw data needs calibration to correct for lens distortions, sensor noise, and camera misalignments, especially in camera arrays or plenoptic systems. Geometric calibration ensures accurate reconstruction.[16]
  • Interpolation and View Synthesis: A key advantage is generating novel viewpoints not explicitly captured. This involves interpolating the 4D light field data to estimate the scene's appearance from arbitrary positions and angles.[17]
  • Depth Estimation: The angular variation of light rays encodes depth information. Various algorithms can extract depth maps, valuable for effects like synthetic depth of field and for AR interactions.[18]
  • Compression: Light field datasets are massive. Efficient compression is vital for storage and transmission, especially for mobile VR/AR and streaming. Techniques often adapt existing video codecs (like VP9) or use specialized approaches.[19][20] Standards bodies like JPEG Pleno and MPEG Immersive Video are developing formats for light field data.[21]

Light Field Rendering and Display

Once captured or generated, light fields enable novel rendering and display possibilities.

Rendering

Rendering involves sampling the 4D light field data to synthesize a 2D image for a virtual camera. For a desired camera pose, the algorithm determines which rays intersect the virtual sensor and integrates their values.[1] This enables:

  • Refocusing: Synthetically changing the focal plane after capture.[3]
  • Depth of Field Control: Computationally adjusting the aperture size.
  • View Synthesis: Generating views from slightly different positions, enabling parallax effects and six degrees of freedom (6DoF) movement.[4]
  • Neural Radiance Fields (NeRF) and Neural Rendering: Modern approaches like NeRF represent the scene as a continuous function learned by a neural network, allowing high-quality view synthesis from sparse inputs. These methods effectively learn a compressed light field representation and are highly relevant for VR/AR rendering.[22][23]

Light Field Displays

Light field displays aim to reproduce the directional aspect of light rays, allowing the viewer's eyes to accommodate (focus) naturally at different depths, potentially resolving the vergence-accommodation conflict inherent in conventional stereoscopic displays.[24][25] This provides more natural depth cues and improves visual comfort. Key approaches include:

Integral Imaging Displays

These use a microlens array placed over a high-resolution display panel (for example OLED, LCD). Each microlens projects pixels underneath it into different directions, creating multiple views of the scene. Densely sampled views approximate a continuous light field, enabling autostereoscopic viewing.[26][27] This is effectively the inverse of a plenoptic camera.

Multi-Plane and Varifocal Displays

These displays present images at multiple discrete focal planes, either simultaneously using stacked transparent displays or rapidly sequentially using varifocal lenses or switchable optics synchronized with eye tracking.[28] While not always full light field displays, they approximate the necessary focus cues. The Light Field Stereoscope is an example of a multi-layer approach.[24][21]

Holographic Displays

Holographic displays reconstruct the light wavefront itself using spatial light modulators (SLMs) to control the phase or amplitude of light. These can, in theory, perfectly reproduce the light field of a scene, offering continuous focus cues.[29] Research includes using Holographic Optical Elements (HOEs) and metasurfaces for compact designs, like Nvidia's Holographic Glasses prototype.[30]

Compressive/Tensor Displays

These use multiple layers of modulating panels (for example LCDs) with computational algorithms to sculpt the light passing through them, synthesizing a target light field with relatively thin hardware.[31][21]

Projector/Pinlight Arrays

Systems using arrays of micro-projectors or scanned beams directed onto specialized screens (for example lenticular sheets), or near-eye displays using arrays of "pinlights" (point sources imaged through microlenses or pinholes) can also generate light fields.[21]

Commercial Examples and Prototypes

Several companies are developing light field or related display technologies:

Applications in VR and AR

Light field technology holds immense promise for VR and AR:

  • Addressing the Vergence-Accommodation Conflict (VAC): This is perhaps the most significant benefit. By providing correct focus cues matching the vergence depth, light field displays can reduce eye strain, nausea, and visual fatigue associated with prolonged use of conventional stereoscopic HMDs, leading to more comfortable and immersive experiences.[25][37]
  • Correct Parallax and 6DoF: Light fields inherently support correct parallax shifts as the user moves their head within the eye-box, crucial for realistic six degrees of freedom (6DoF) experiences in VR and stable anchoring of virtual objects in AR.[4] Google's "Welcome to Light Fields" VR app demonstrated this effect.[39]
  • Enhanced Visual Fidelity and View-Dependent Effects: Light fields capture and reproduce complex light interactions like specular highlights, transparency, reflections, and refractions more accurately than traditional rendering, enhancing realism.[22]
  • Realistic Capture for VR/AR Content: Light field cameras capture real-world scenes with richer information than 360° video or basic photogrammetry, preserving subtle lighting and allowing more natural exploration in VR. Systems like Google's light field capture rigs and Lytro Immerge were developed for this.[39]
  • Light Field Passthrough for Mixed Reality: An emerging technique for AR/MR headsets where specialized cameras capture the light field of the real world. This allows rendering the outside view with correct depth and perspective for the user's eyes, enabling seamless blending of virtual objects with reality and minimizing reprojection errors or distortions seen in traditional video passthrough. Meta's Flamera prototype is a notable example.[40][41][42]
  • Telepresence and Remote Collaboration: Realistic capture and display of participants using light fields can significantly enhance the sense of presence in virtual meetings and remote collaboration systems, enabling more natural eye contact and spatial interaction.[43]
  • Post-Capture Refocus and DoF Control: While primarily a photographic benefit, this capability could be used in VR/AR for cinematic effects, accessibility features, or interactive storytelling.

Advantages

  • Provides highly realistic views with correct parallax and view-dependent effects.
  • Enables post-capture refocusing and depth of field adjustments (primarily capture advantage).
  • Potential to significantly reduce or eliminate the vergence-accommodation conflict in HMDs, increasing comfort.
  • Captures rich scene information useful for various computational photography and computer vision tasks (for example depth estimation).
  • Enables more seamless integration of virtual elements in AR/MR via techniques like light field passthrough.

Challenges and Limitations

  • Data Size and Management: 4D light fields represent significantly more data than conventional images, posing challenges for capture, storage (often terabytes for high quality), transmission (streaming), and processing.[44]
  • Computational Complexity: Processing and rendering light fields, especially in real-time for high-resolution VR/AR, requires substantial computational power. Optimization and machine learning approaches are active research areas.[45]
  • Capture Hardware Complexity and Cost: High-quality light field capture systems (plenoptic cameras, large camera arrays) remain complex, expensive, and often limited to controlled environments.[46]
  • Display Technology Immaturity and Trade-offs: High-performance light field displays suitable for consumer VR/AR HMDs (high resolution, high brightness, wide field of view (FoV), large eye-box, low latency, compact form factor) are still largely under development. Current technologies often involve trade-offs, for example between spatial and angular resolution.[47][21]
  • Limited Angular Resolution: Practical systems often have limited angular resolution, which can restrict the range of parallax and the effectiveness in fully resolving VAC.
  • Eye-Box Size: Some display approaches (especially holographic and integral imaging) can have a limited viewing zone (eye-box) where the effect is perceived correctly, requiring precise alignment or eye tracking compensation.[21]

Future Directions

Research and development continue to advance light field technology:

  • Neural Radiance Fields (NeRF) and Neural Rendering: These machine learning techniques are rapidly evolving, offering efficient ways to represent and render complex scenes with view-dependent effects, potentially revolutionizing light field capture and synthesis for VR/AR.[22]
  • Eye-Tracked Foveated Light Fields: Combining eye tracking with light field rendering/display allows concentrating detail and computational resources where the user is looking (foveated rendering), making real-time performance more feasible.[48][49]
  • Compact Light Field Optics: Development of metalenses, diffractive optics, novel waveguide designs, and HOEs aims to create thinner, lighter, and more efficient optics for near-eye light field displays suitable for glasses-like AR/VR devices.[50][30]
  • Light Field Video Streaming: Advances in compression and network bandwidth may enable real-time streaming of light field video for immersive communication, entertainment, and training.[51]
  • Integration with Haptics and Spatial Audio: Combining high-fidelity light field visuals with haptic feedback and spatial audio promises truly immersive multisensory experiences.[52]

See Also

References

  1. ↑ Jump up to: 1.0 1.1 1.2 1.3 1.4 Levoy, M., & Hanrahan, P. (1996). Light field rendering. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96, 31-42.
  2. ↑ Jump up to: 2.0 2.1 2.2 2.3 Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996). The Lumigraph. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96, 43-54.
  3. ↑ Jump up to: 3.0 3.1 3.2 3.3 Ng, R. (2005). Digital Light Field Photography. Ph.D. Thesis, Stanford University.
  4. ↑ Jump up to: 4.0 4.1 4.2 Lanman, D., & Luebke, D. (2013). Near-eye light field displays. ACM SIGGRAPH 2013 Talks, 1-1.
  5. ↑ Faraday, M. (1846). Thoughts on Ray Vibrations. Philosophical Magazine, S.3, Vol. 28, No. 188.
  6. ↑ Gershun, A. (1939). The Light Field. Journal of Mathematics and Physics, 18(1-4), 51-151. (English translation of 1936 Russian paper).
  7. ↑ Jump up to: 7.0 7.1 7.2 Wikipedia: Light field
  8. ↑ Jump up to: 8.0 8.1 Adelson, E. H., & Bergen, J. R. (1991). The plenoptic function and the elements of early vision. In Computational Models of Visual Processing (pp. 3-20). MIT Press.
  9. ↑ Stanford University: Light fields and computational photography
  10. ↑ Jump up to: 10.0 10.1 Wikipedia: Light field camera
  11. ↑ Adelson, E. H., & Wang, J. Y. A. (1992). Single lens stereo with a plenoptic camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 99-106.
  12. ↑ Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., ... & Levoy, M. (2005). High performance imaging using large camera arrays. ACM Transactions on Graphics (TOG), 24(3), 765-776.
  13. ↑ Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., Adams, A., Horowitz, M., & Levoy, M. (2005). "High performance imaging using large camera arrays." ACM Transactions on Graphics, 24(3), 765-776.
  14. ↑ Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007). Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Transactions on Graphics (TOG), 26(3), 69-es.
  15. ↑ Moon, B., McDermott, S., Golas, K., Cao, Y., & Wong, T.-T. (2016). "Efficient rendering of virtual objects for mixed reality." Computers & Graphics, 57, 93-101.
  16. ↑ Bok, Y., Jeon, H. G., & Kweon, I. S. (2017). "Geometric calibration of micro-lens-based light field cameras using line features." IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(2), 287-300.
  17. ↑ Kalantari, N. K., Wang, T. C., & Ramamoorthi, R. (2016). "Learning-based view synthesis for light field cameras." ACM Transactions on Graphics, 35(6), 193.
  18. ↑ Tao, M. W., Hadap, S., Malik, J., & Ramamoorthi, R. (2013). "Depth from combining defocus and correspondence using light-field cameras." Proceedings of the IEEE International Conference on Computer Vision, 673-680.
  19. ↑ Viola, I., Rerabek, M., & Ebrahimi, T. (2017). "Comparison and evaluation of light field image coding approaches." IEEE Journal of Selected Topics in Signal Processing, 11(7), 1092-1106.
  20. ↑ Augmented Perception: Welcome to Light Fields
  21. ↑ Jump up to: 21.0 21.1 21.2 21.3 21.4 21.5 IEEE ComSoc MMTC Review Letter, Vol. 8, No. 2, Nov 2017
  22. ↑ Jump up to: 22.0 22.1 22.2 Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). "NeRF: Representing scenes as neural radiance fields for view synthesis." European Conference on Computer Vision, 405-421.
  23. ↑ VR-NeRF Project Page
  24. ↑ Jump up to: 24.0 24.1 Huang, F. C., Chen, K., & Wetzstein, G. (2015). "The light field stereoscope: Immersive computer graphics via factored near-eye light field displays with focus cues." ACM Transactions on Graphics, 34(4), 60.
  25. ↑ Jump up to: 25.0 25.1 Konrad, R., Cooper, E. A., Wetzstein, G., & Banks, M. S. (2017). Accommodation and vergence responses to near-eye light field displays. Journal of Vision, 17(10), 987-987.
  26. ↑ Martinez-Corral, M., & Javidi, B. (2018). "Fundamentals of 3D imaging and displays: A tutorial on integral imaging, light-field, and plenoptic systems." Proceedings of the IEEE, 106(5), 891-908.
  27. ↑ Jones, A., McDowall, I., Yamada, H., Bolas, M., & Debevec, P. (2007). Rendering for an interactive 360° light field display. ACM Transactions on Graphics (TOG), 26(3), 40-es.
  28. ↑ Mercier, O., Sulai, Y., Mackenzie, K., Zannoli, M., Hillis, J., Nowrouzezahrai, D., & Lanman, D. (2017). "Fast gaze-contingent optimal decompositions for multifocal displays." ACM Transactions on Graphics, 36(6), 237.
  29. ↑ Li, G., Lee, D., Jeong, Y., Cho, J., & Lee, B. (2016). "Holographic display for see-through augmented reality using mirror-lens holographic optical element." Optics Letters, 41(11), 2486-2489.
  30. ↑ Jump up to: 30.0 30.1 Nvidia Developer Blog: Holographic Glasses Research
  31. ↑ Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2011). Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2011, 30(4), 95:1-95:12.
  32. ↑ Looking Glass Factory. (2021). "Technical Overview of Looking Glass Holographic Displays." Looking Glass Technical Documentation.
  33. ↑ Cheng, D., Wang, Q., Liu, Y., & Wang, J. (2017). "Retinal projection type super multi-view 3D head-mounted display." Proceedings of SPIE Optical Engineering + Applications, 10335.
  34. ↑ Kress, B. C., & Chatterjee, I. (2020). "Waveguide combiners for mixed reality headsets: a nanophotonics design perspective." Nanophotonics, 9(11), 3653-3667.
  35. ↑ fxguide: Light Fields - The Future of VR-AR-MR
  36. ↑ Fattal, D., Peng, Z., Tran, T., Vo, S., Fiorentino, M., Brug, J., & Beausoleil, R. G. (2013). "A multi-directional backlight for a wide-angle, glasses-free three-dimensional display." Nature, 495(7441), 348-351.
  37. ↑ Jump up to: 37.0 37.1 Road to VR: Hands-on: CREAL's Light-field Display Brings a New Layer of Immersion to AR
  38. ↑ CREAL: Light-field Display Technology
  39. ↑ Jump up to: 39.0 39.1 Google Blog: Experimenting with Light Fields
  40. ↑ Tekno Asian: Light Field Passthrough: The Bridge Between Reality and Virtual Worlds
  41. ↑ IEEE Spectrum: Meta Builds AR Headset With Unrivaled Passthrough
  42. ↑ Display Daily: Meta’s Perspective-Correct Passthrough MR Display
  43. ↑ Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W., Kowdle, A., Degtyarev, Y., Kim, D., Davidson, P. L., Khamis, S., Dou, M., Tankovich, V., Loop, C., Cai, Q., Chou, P. A., Mennicken, S., Valentin, J., Pradeep, V., Wang, S., Kang, S. B., Kohli, P., Lutchyn, Y., Keskin, C., & Izadi, S. (2016). "Holoportation: Virtual 3D teleportation in real-time." Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 741-754.
  44. ↑ Alain, M., Smolic, A., & Guillemot, C. (2019). "Light field compression: A review." IEEE Journal of Selected Topics in Signal Processing, 13(3), 454-463.
  45. ↑ Wang, T. C., Efros, A. A., & Ramamoorthi, R. (2021). "Neural rendering and neural light transport for mixed reality." IEEE Transactions on Visualization and Computer Graphics, 27(5), 2657-2671.
  46. ↑ Overbeck, R. S., Erickson, D., Evangelakos, D., Pharr, M., & Debevec, P. (2018). "A system for acquiring, processing, and rendering panoramic light field stills for virtual reality." ACM Transactions on Graphics, 37(6), 197.
  47. ↑ Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2012). "Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting." ACM Transactions on Graphics, 31(4), 80.
  48. ↑ Kaplanyan, A. S., Sochenov, A., Leimkühler, T., Okunev, M., Goodall, T., & Rufo, G. (2019). "DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos." ACM Transactions on Graphics, 38(6), 212.
  49. ↑ Nvidia Research: Foveated Light-field Rendering
  50. ↑ Wang, N., Hua, H., & Viegas, D. (2021). "Compact optical see-through head-mounted display with varifocal liquid membrane lens." Digital Holography and Three-Dimensional Imaging 2021, OSA Technical Digest, DM3B.3.
  51. ↑ Gutiérrez-Navarro, D., & Pérez-Daniel, K. R. (2022). "Light field video streaming: A review." IEEE Access, 10, 12345-12367.
  52. ↑ Martins, R., Terziman, L., Paljic, A., & Pettré, J. (2021). "Integrating vision, haptics and proprioception into a unified 3D perception framework for immersive virtual reality." IEEE Transactions on Haptics, 14(2), 401-410.
Retrieved from "https://vrarwiki.com/index.php?title=Light_field&oldid=34694"