Multifocal display
More actions
A multifocal display (also called a multi-focal-plane or multi-plane display) is a 3D display, often built as a near-eye display, that shows imagery at several fixed focal distances instead of one, so the viewer's eyes can focus at different depths in the virtual scene. In their 2020 review of the field, Tao Zhan, Jianghao Xiong, Junyu Zou and Shin-Tson Wu describe multifocal displays as systems that "generate multiple 2D virtual images spanning the desired range of spatial focal depths", giving a stack of focal planes that provides approximately correct depth blur.[1]
The technique is one of several proposed remedies for the vergence-accommodation conflict (VAC) in head-mounted displays. A conventional stereoscopic headset drives vergence correctly through binocular disparity, but the eye's focus (accommodation) stays fixed at the single distance set by the panel and lens. Zhan et al. note that this mismatch can slow binocular processing and lower the precision of depth perception, and that it has been reported to cause eye strain, blurred vision and headache.[1] Multifocal designs have been proposed and prototyped since the 1990s, but the same review observed that the technology "is rarely implemented in current commercial display products".[1] One commercial example is the Magic Leap One, which used two focal planes per eye; its successor, the Magic Leap 2, went back to a single focal plane.[2][3]
How it works
In natural vision the eyes converge on an object and the lens of each eye focuses at the same distance. A stereoscopic display reproduces the convergence cue but not the focus cue, because all light reaches the eye from the optical image of one panel.[1] A multifocal display adds physical depth: it forms two or more images whose virtual distances are spread through the range the eye needs to focus over, and draws each part of the scene on the plane, or pair of planes, nearest to its intended depth. The light really comes from different focal distances, so the eye receives near-correct accommodation stimuli and natural retinal blur without needing to know where the user is looking.[1][4]
Akeley, Watt, Girshick and Banks called their 2004 prototype a pair of "fixed-viewpoint volumetric displays": like other volumetric displays, they generate near-correct focus cues without tracking eye position, "because light comes from sources at the correct focal distances", but unlike autostereoscopic volumetric displays they keep view-dependent effects such as occlusion, specularity and reflection, and can use ordinary graphics processors and 2D display panels.[4]
Depth-weighted blending
A small number of planes cannot hold a continuous scene, so content lying between two planes is split across both. In the scheme Akeley and colleagues used, each point is drawn on the two nearest planes, with the intensity on each plane weighted by the point's dioptric distance from that plane; the two weights add up to the brightness a single plane would show. Hoffman and co-authors wrote that this rule "eliminates the discontinuity" in blur that assigning each point to only its nearest plane would produce.[5] The technique is also called depth filtering[6] or depth fusion.[7] Sheng Liu and Hong Hua proposed a set of nonlinear depth-weighted fusing functions in 2010;[7] a 2011 comparison by Sowmya Ravikumar, Akeley and Banks, using a model of a typical human eye, concluded that the linear rule of Akeley and colleagues "is the best solution for natural stimuli".[8]
Simple blending produces artifacts at occlusion boundaries and with reflective or refractive surfaces.[1] Rahul Narain and colleagues (2015) instead computed the images for each plane with an optimization that uses a model of defocus in the viewer's eye, so that the retinal images formed while focusing at different distances match those of the original scene as closely as possible; the method handles occlusions and non-Lambertian effects and needs no explicit scene geometry.[9] Zhan et al. note that this approach was still too slow for real-time frame rates.[1] At SIGGRAPH Asia 2017, Olivier Mercier, Douglas Lanman and co-authors presented a GPU-friendly version of optimal decomposition with what they reported as a "three-orders-of-magnitude speedup over previous work", showed that eye tracking can keep the image planes aligned with the eyes through image-based deformations, and built what they called "the first binocular multifocal testbed with integrated eye tracking and accommodation measurement".[10][11]
How many planes, and how far apart
Plane spacing is set by the eye's depth of focus. Jannick Rolland, Myron Krueger and Alexei Goon calculated in 1999 that, for a visual acuity of one arcminute, 27 planes were needed to cover accommodation from infinity to 0.5 m, and that 14 could suffice because every other plane can serve as a plane of accommodation.[12] Their 2000 journal paper in Applied Optics stated that the display "requires at most 27 equally spaced planes", in a stack about 26 mm wide with in-plane resolution of the order of 5 micrometres.[13]
Later work found that blending allows much sparser stacks. Liu and Hua concluded in 2010 that adjacent planes should be about 0.6 diopters apart, with smaller spacing possibly needed to improve retinal image quality further.[7] In an accommodation study, Kevin MacKenzie, David Hoffman and Simon Watt measured focusing responses to depth-filtered stimuli and found them "accurate and precise for image-plane separations up to ~1 diopter"; at larger separations responses broke down.[6] The practical cost is bandwidth: every added plane needs its own image, delivered through more panels, more frames per second, or another channel of light.[1]
Optical architectures
Zhan et al. classify multifocal designs in two ways. By how depth is produced, they are distance-based, where several display panels or screens sit at different optical path lengths, or power-based, where an optical element changes its focal power on command or according to properties of the incoming light. By how the extra image information is carried, they are space-, time-, polarization- or wavelength-multiplexed, or hybrids of these.[1]
| Multiplexing channel | How the planes are produced | Main trade-off (per Zhan et al.) |
|---|---|---|
| Space | Stacked transparent displays or screens, or several panels combined with stacked beam splitters, freeform prisms or light guides | Larger volume or reduced resolution; transparent panels lose brightness and can show diffraction and Moire effects[1] |
| Time | One fast display synchronized with a tunable or switchable lens, a deformable mirror or a stack of switchable diffuser screens, showing each depth in turn faster than flicker can be seen | Needs high-refresh displays and fast optics; frame rate is divided among the planes[1] |
| Polarization | Pixels encode polarization state, and a polarization-dependent lens sends each state to a different depth | Limited to two independent focal depths, because there are only two orthogonal polarization states[1] |
| Wavelength | Narrow-band light sources at slightly different wavelengths are separated into different depths by a spectrally selective element | Reduces color gamut, since only colors shared by all planes can be shown uniformly[1] |
Zhan et al. found that most reported designs use time multiplexing, trading display frame rate for more focal planes, and expected that trend to continue because of its compact form factor.[1] Power-switching optics used in these designs include electrowetting and fluid-membrane liquid lenses, birefringent crystal lenses, liquid crystal lenses and switchable Pancharatnam-Berry phase lenses.[1] In a 2021 review in Light: Science and Applications, Xiong and co-authors explain that stacking N actively switchable Pancharatnam-Berry lenses can produce 2N depths, and that waveguide combiners with grating out-couplers, which form an image at a fixed depth at infinity, generally need extra stacked waveguide layers to add depths.[14]
Relation to other focus-supporting displays
A varifocal display also moves the image depth, but it shows only one depth at a time and uses eye tracking to place that plane where the eyes are verging.[14] Zhan et al. describe the difference this way: varifocal displays "actively track the accommodation of the eye in real time", while multifocal displays "passively" offer several depths without regard to where the eye focuses.[1] Because the two share hardware, Xiong et al. note that a varifocal display "can often be converted to a multifocal display" if its focus module is fast enough to show several depths within one frame.[14]
A focal surface display, introduced by Nathan Matsuda, Alexander Fix and Lanman in 2017, places a phase-only spatial light modulator between the panel and the eyepiece so that it acts as a dynamic freeform lens, bending the focal surfaces to follow the scene geometry instead of keeping them flat; the authors described it as building on "optimized blending" to implement a multifocal display.[15] Holographic displays aim to reconstruct the real wavefront of a scene, while Maxwellian view retinal projection gives a blur-free image whatever the eye's focusing state, so it cannot reproduce natural optical blur. Zhan et al. regard holographic and volumetric approaches as not yet practical because of their demanding hardware requirements.[1]
History and research prototypes
Rolland, Krueger and Goon examined a head-mounted display based on multiple focal plane arrays in a 1999 SPIE paper, and in their 2000 Applied Optics paper proposed one built around a stack of laminated planes.[12][13] In 2004 Akeley and colleagues built a working stereo prototype with three image planes per eye. Zhan et al. describe it as an LCD panel divided into three sub-panels by stacked beam splitters, with planes 0.67 diopters apart, and call it more a proof of concept than a practical implementation because of its size.[4][1] Using a three-plane display at focal distances of 31.1, 39.4 and 53.6 cm (a spacing the authors described as "just slightly greater than a standard observer's depth of focus"), Hoffman, Girshick, Akeley and Banks showed in 2008 that when focus cues were correct or nearly correct, viewers identified stereoscopic stimuli faster, had better stereoacuity in a time-limited task, saw less depth distortion, and reported less fatigue and discomfort.[5]
Further milestones include:
| Year | Researchers | System | Reported characteristics |
|---|---|---|---|
| 2004 | Alan Sullivan (LightSpace Technologies) | DepthCube, a direct-view multiplanar display: a DLP projector with an air-spaced stack of 20 switchable liquid crystal scattering screens | 1024 x 768 images in 20 depths at 50 Hz, 5-bit color[1] |
| 2008 | Sheng Liu, Dewen Cheng, Hong Hua | Optical see-through HMD with addressable focal planes, using an electrowetting liquid lens in a birdbath layout | Dual-focal operation limited to about 7 Hz by the lens response; follow-up versions raised the dual-focal rate to 37.5 Hz (18.75 Hz with blank frames between images) and then to 21.25 Hz with blank frames[16][1] |
| 2009 | Gordon Love, David Hoffman, Martin Banks and others | Stereoscopic display with a fast switchable lens synchronized to the display | Four focal planes from two birefringent calcite lenses and a ferroelectric liquid crystal polarization switch, shown at 45 Hz from a 180 Hz CRT[17][1] |
| 2011-2014 | Xinda Hu, Hong Hua | Digital Micromirror Device display with a deformable membrane mirror, later paired with a see-through freeform prism eyepiece | Six focal planes spanning 0 to 3 diopters; the 2014 see-through version reported 1.8 arcminute angular resolution across a 40-degree diagonal field of view[18][1] |
| 2015 | Patrick Llull and co-authors | Binocular benchtop display with a 400 Hz DMD and a fast fluid-membrane tunable lens | Six focal planes 0.6 diopters apart at 60 Hz with a 31-degree field of view[1] |
| 2018 | Tao Zhan, Yun-Han Lee, Shin-Tson Wu | HMD using two switchable liquid crystal Pancharatnam-Berry lenses with a 240 Hz LCD | Four depths at 60 Hz in a form factor similar to commercial VR headsets[1] |
| 2018 | Jen-Hao Rick Chang, B. V. K. Vijaya Kumar, Aswin Sankaranarayanan | Near-eye display with a continuously sweeping focus-tunable lens whose focal length is tracked optically | 1,600 focal planes per second, 40 depths at 40 Hz, monochrome[19][1] |
| 2018 | Kishore Rathinavel, Hanpeng Wang, Alex Blate, Henry Fuchs | Full-color near-eye AR display with a sweeping tunable lens, high-speed DMD projector and HDR light source | 280 binary image depths from 15 cm (6.7 diopters) to 4 m (0.25 diopters)[20] |
| 2019 | Roberts Zabels and others (LightSpace Technologies) | HMD with stacked liquid crystal diffuser elements and a synchronized high-refresh projector | Six image depth planes, 72-degree horizontal field of view, perceptually continuous accommodation over a 3.2 diopter range[21][1] |
Some research systems blur the line with other categories. The tomographic near-eye display from Seoul National University (2019) pairs a 60 Hz LCD with a DMD backlight and a tunable lens so that each pixel is lit only when the sweeping image plane reaches its depth, giving 80 depths between 0 and 5.5 diopters; Zhan et al. note it cannot show semi-transparent objects at several depths along one line of sight, since each LCD pixel is fixed at one depth.[1]
Use in VR and AR products
In its 2018 teardown of the Magic Leap One, a mixed reality headset from Magic Leap, iFixit found that "each eye gets two sets of red, green, and blue screen layers that create two focal planes", delivered through six-layer waveguides.[2] In his June 2022 hands-on with the Magic Leap 2, Road to VR's Ben Lang reported that the "photonic lightfield chip" of the first model ("AKA a display with two focal planes") was gone, and that the new headset "only supports a single focal plane (as is the case for pretty much all AR headsets available today)".[3]
Avegant announced a "Light Field Technology" for mixed reality in March 2017, stating that it "enables the visualization of objects at multiple focal planes for augmented and mixed reality applications".[22]
LightSpace Technologies, the Latvian company behind the 2019 six-plane prototype, signed a partnership with Rheinmetall Electronics UK in June 2024 to develop and produce multi-focal AR systems for defence and security use, following proof-of-concept work in 2023. The companies said they planned to launch the productised version, ACE-AR, at the UK Ministry of Defence's Defence Vehicle Demonstration exhibition later that year.[23][24]
The 2017 multifocal rendering paper by Mercier, Lanman and colleagues is listed among Meta's research publications, but the focus-supporting headset Meta's Reality Labs showed at SIGGRAPH 2023, Butterscotch Varifocal, is a varifocal design: it uses eye tracking and moves the display closer to or farther from the eyes. In the same post Meta said that current headsets are "limited by a fixed focal distance".[10][25]
Limitations
The review by Zhan et al. lists the main obstacles. Space-multiplexed stacks grow bulky or lose resolution, and transparent displays trade brightness for transparency. Time multiplexing divides the frame rate among the planes at a time when headsets already need high refresh rates to reduce motion blur, and it requires fast-switching optics. Polarization multiplexing tops out at two planes, and wavelength multiplexing narrows the color gamut.[1] Rendering adds its own cost, since optimized decompositions that handle occlusion correctly are much heavier to compute than simple blending, and Mercier et al. noted that such decompositions "likely require better alignment of the image planes with the viewer's eyes".[10] For AR waveguides, adding depths usually means stacking more waveguide layers on top of the separate layers already used for red, green and blue, which increases thickness.[14]
See also
References
- ↑ 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 1.20 1.21 1.22 1.23 1.24 1.25 1.26 1.27 Tao Zhan, Jianghao Xiong, Junyu Zou, Shin-Tson Wu (2020). "Multifocal displays: review and prospect". PhotoniX, vol. 1, article 10. https://doi.org/10.1186/s43074-020-00010-0. Retrieved 2026-10-15.
- ↑ 2.0 2.1 Sam Goldheart (2018-08-22). "The Science Behind the Magic Leap". iFixit. https://www.ifixit.com/News/magic-leap-teardown. Retrieved 2026-10-15.
- ↑ 3.0 3.1 Ben Lang (2022-06-10). "Hands-on: Magic Leap 2 Shows Clear Improvements, But HoloLens 2 Retains Some Advantages". Road to VR. https://www.roadtovr.com/magic-leap-2-hands-on-vs-hololens-2-impressions/. Retrieved 2026-10-15.
- ↑ 4.0 4.1 4.2 Kurt Akeley, Simon J. Watt, Ahna Reza Girshick, Martin S. Banks (2004-08). "A stereo display prototype with multiple focal distances". ACM Transactions on Graphics, vol. 23, no. 3, pp. 804-813 (SIGGRAPH 2004). https://doi.org/10.1145/1015706.1015804. Retrieved 2026-10-15.
- ↑ 5.0 5.1 David M. Hoffman, Ahna R. Girshick, Kurt Akeley, Martin S. Banks (2008-03-28). "Vergence-accommodation conflicts hinder visual performance and cause visual fatigue". Journal of Vision, vol. 8, no. 3, article 33. doi:10.1167/8.3.33. https://pmc.ncbi.nlm.nih.gov/articles/PMC2879326/. Retrieved 2026-10-15.
- ↑ 6.0 6.1 K. J. MacKenzie, D. M. Hoffman, S. J. Watt (2010-07-26). "Accommodation to multiple-focal-plane displays: Implications for improving stereoscopic displays and for accommodation control". Journal of Vision, vol. 10, no. 8, article 22. https://doi.org/10.1167/10.8.22. Retrieved 2026-10-15.
- ↑ 7.0 7.1 7.2 Sheng Liu, Hong Hua (2010-05-17). "A systematic method for designing depth-fused multi-focal plane three-dimensional displays". Optics Express, vol. 18, no. 11, pp. 11562-11573. https://doi.org/10.1364/OE.18.011562. Retrieved 2026-10-15.
- ↑ Sowmya Ravikumar, Kurt Akeley, Martin S. Banks (2011-10-06). "Creating effective focus cues in multi-plane 3D displays". Optics Express, vol. 19, no. 21, pp. 20940. https://doi.org/10.1364/OE.19.020940. Retrieved 2026-10-15.
- ↑ Rahul Narain, Rachel A. Albert, Abdullah Bulbul, Gregory J. Ward, Martin S. Banks, James F. O'Brien (2015-07-27). "Optimal presentation of imagery with focus cues on multi-plane displays". ACM Transactions on Graphics, vol. 34, no. 4 (SIGGRAPH 2015). https://doi.org/10.1145/2766909. Retrieved 2026-10-15.
- ↑ 10.0 10.1 10.2 Olivier Mercier, Yusufu Sulai, Kevin Mackenzie, Marina Zannoli, James Hillis, Derek Nowrouzezahrai, Douglas Lanman (2017-11-20). "Fast gaze-contingent optimal decompositions for multifocal displays". ACM Transactions on Graphics, vol. 36, no. 6 (SIGGRAPH Asia 2017). https://doi.org/10.1145/3130800.3130846. Retrieved 2026-10-15.
- ↑ "Fast Gaze-Contingent Optimal Decompositions for Multifocal Displays". Meta Research. https://research.facebook.com/publications/fast-gaze-contingent-optimal-decompositions-for-multifocal-displays/. Retrieved 2026-10-15.
- ↑ 12.0 12.1 Jannick P. Rolland, Myron W. Krueger, Alexei A. Goon (1999-01). "Dynamic focusing in head-mounted displays". Proceedings of SPIE, vol. 3639 (The Engineering Reality of Virtual Reality 1999), pp. 463-470. https://api.creol.ucf.edu/Publications/1484.pdf. Retrieved 2026-10-15.
- ↑ 13.0 13.1 Jannick P. Rolland, Myron W. Krueger, Alexei Goon (2000-07-01). "Multifocal planes head-mounted displays". Applied Optics, vol. 39, no. 19, pp. 3209-3215 (record in UCF STARS). doi:10.1364/AO.39.003209. https://stars.library.ucf.edu/facultybib2000/2776. Retrieved 2026-10-15.
- ↑ 14.0 14.1 14.2 14.3 Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan, Shin-Tson Wu (2021-10-25). "Augmented reality and virtual reality displays: emerging technologies and future perspectives". Light: Science and Applications, vol. 10, article 216. https://doi.org/10.1038/s41377-021-00658-8. Retrieved 2026-10-15.
- ↑ Nathan Matsuda, Alexander Fix, Douglas Lanman (2017-07-20). "Focal surface displays". ACM Transactions on Graphics, vol. 36, no. 4 (SIGGRAPH 2017). https://doi.org/10.1145/3072959.3073590. Retrieved 2026-10-15.
- ↑ Sheng Liu, Dewen Cheng, Hong Hua (2008-09). "An optical see-through head mounted display with addressable focal planes". 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp. 33-42. https://doi.org/10.1109/ISMAR.2008.4637321. Retrieved 2026-10-15.
- ↑ Gordon D. Love, David M. Hoffman, Philip J. W. Hands, James Gao, Andrew K. Kirby, Martin S. Banks (2009-08-20). "High-speed switchable lens enables the development of a volumetric stereoscopic display". Optics Express, vol. 17, no. 18, pp. 15716. https://doi.org/10.1364/OE.17.015716. Retrieved 2026-10-15.
- ↑ Xinda Hu, Hong Hua (2014-05-30). "High-resolution optical see-through multi-focal-plane head-mounted display using freeform optics". Optics Express, vol. 22, no. 11, pp. 13896. https://doi.org/10.1364/OE.22.013896. Retrieved 2026-10-15.
- ↑ Jen-Hao Rick Chang, B. V. K. Vijaya Kumar, Aswin C. Sankaranarayanan (2018-12-04). "Towards multifocal displays with dense focal stacks". ACM Transactions on Graphics, vol. 37, no. 6 (SIGGRAPH Asia 2018). https://doi.org/10.1145/3272127.3275015. Retrieved 2026-10-15.
- ↑ Roberts Zabels, Kriss Osmanis, Martins Narels, Ugis Gertners, Ainars Ozols, Karlis Rutenbergs, Ilmars Osmanis (2019-08-02). "AR Displays: Next-Generation Technologies to Solve the Vergence-Accommodation Conflict". Applied Sciences, vol. 9, no. 15, article 3147. https://doi.org/10.3390/app9153147. Retrieved 2026-10-15.
- ↑ Avegant (2017-03-15). "Avegant Introduces Light Field Technology for Mixed Reality". PR Newswire. https://www.prnewswire.com/news-releases/avegant-introduces-light-field-technology-for-mixed-reality-300423855.html. Retrieved 2026-10-15.
- ↑ Shephard News Team (2024-06-14). "Rheinmetall and Lightspace Technologies to develop augmented reality systems". Shephard Media. https://www.shephardmedia.com/news/training-simulation/rheinmetall-and-lightspace-technologies-to-develop-augmented-reality-systems/. Retrieved 2026-10-15.
- ↑ "Rheinmetall Electronics UK and Lightspace Technologies Sign Partnership Agreement". Defence Turkey. 2024-08-08. https://defenceturkey.com/news/rheinmetall-electronics-uk-and-lightspace-technologies-sign-partnership-agreement. Retrieved 2026-10-15.
- ↑ "Demo or Die: How Reality Labs' Display Systems Research Team Is Pushing the VR Industry Toward the Future". Meta. 2023-07-31. https://www.meta.com/blog/reality-labs-research-display-systems-siggraph-2023-butterscotch-varifocal-flamera/. Retrieved 2026-10-15.