Gaze-contingent display
More actions
A gaze-contingent display (GCD) is a display system whose content changes in real time according to where the viewer is looking, as measured by an eye tracker. In their 2004 review, Andrew Duchowski, Nathan Cournia and Hunter Murphy describe gaze-contingent displays as an attempt "to balance the amount of information displayed against the visual information processing capacity of the observer through real-time eye movement sensing".[1] A common type centres high-resolution imagery on the point of gaze and lowers resolution elsewhere, which is the principle behind gaze-tracked foveated rendering in current head-mounted displays.[2][3]
The approach was used in reading research by the mid-1970s. In 1975 George McConkie and Keith Rayner introduced the "moving window" paradigm, in which text outside a window around the reader's fixation was masked and the window moved with the eyes.[4][5] Engineers have used the same idea to reduce image bandwidth and rendering work in flight simulators, virtual reality, teleoperation, image transmission and video teleconferencing.[2] In VR and augmented reality (AR), gaze-contingent methods are now used to speed up rendering, to drive varifocal optics, to render depth of field and ocular parallax, and to redirect a walking user during saccades.[3][6][7]
Definition and terminology
A gaze-contingent display combines two components: a system that measures the viewer's point of gaze (by combined eye and head tracking) and a display whose image can be modified in real time in response.[2] Andrew Duchowski's taxonomy of gaze-based interaction places gaze-contingent displays in the "passive" category, alongside diagnostic (off-line measurement), active (selection) and expressive (gaze synthesis) uses. Passive interaction, in his definition, is "the manipulation of scene elements in response to gaze direction", for example to improve frame rate; the user does not select anything with the eyes, and the system responds to gaze continuously.[3]
Duchowski, Cournia and Murphy divide gaze-contingent displays into three groups by what they change:[1]
| Type | What is changed in response to gaze |
|---|---|
| Screen-based displays | Pixel-level information, generally matched to the resolvability of the human retina to make efficient use of bandwidth |
| Model-based displays | Geometric primitives of a 3D scene, with similar goals |
| Attentive user interfaces | Object-level entities such as windows and applications, depending on the assumed attentive state of the user |
Duchowski's 2018 survey similarly distinguishes model-based foveation, which simplifies scene geometry (for example, rendering fewer triangles outside the foveal region), from pixel-based foveation, which reduces the spatial or temporal detail of pixel data just before rendering.[3]
The field uses many overlapping terms. Eyal Reingold, Lester Loschky, George McConkie and David Stampe note in their 2003 review in Human Factors that the gaze-contingent aspect of these displays has also been called "foveated" or "eye-slaved", that the multiresolution aspect has been called "variable resolution", "space variant", "area of interest" or "level of detail", and that the combination is sometimes simply called a "moving window".[2] The authors separate the attended area of interest (the region around the point of gaze that the viewer is attending to) from the displayed area of interest (the high-resolution region in the image), and they distinguish displays with a discrete drop-off (for example a biresolutional image with one high and one low level) from those with a continuous drop-off in resolution away from the gaze point.[2]
Rationale
The human retina resolves fine detail only in a small central region, the fovea, and acuity falls with eccentricity. Reingold and colleagues put the maximum resolution of the visual system at "at least 60 cycles/degree or 120 pixels/degree" and note that it is not feasible to build image generators, cameras or display systems that cover the entire field of view with the resolution of the foveal region.[2] A gaze-contingent multiresolutional display places the high-resolution region where the viewer is looking and degrades the rest, which reduces the bandwidth needed to transmit images and, for computer-generated images, the processing needed to render them.[2] Eye movements also help: perceptual thresholds are raised just before, during and after a saccade (saccadic suppression), which helps mask the image change that accompanies moving the high-resolution region to a new fixation.[2]
History
Reading research
Gaze-contingent display changes were in use as research tools in perception studies by the 1970s. A 1973 paper by Stephen M. Reder in Behavior Research Methods & Instrumentation discussed on-line monitoring of eye-position signals for contingent and noncontingent experimental paradigms.[8] In 1975 McConkie and Rayner introduced the moving window paradigm, in which the amount of text available on each fixation is manipulated directly "by changing the display as a function of eye position": text inside the window is shown normally and text outside it is mutilated, for example replaced with x's.[4][5] If reading slows when the window is small, the masked letters were information the reader normally uses. Studies with the paradigm found that the perceptual span in reading is asymmetric, reaching about 3 to 4 letters to the left of fixation and about 14 to 15 letters to the right for adult readers.[5][9]
Related display-change techniques followed:
| Paradigm | Introduced | How the display changes |
|---|---|---|
| Moving window | McConkie and Rayner (1975)[4] | Text outside a window around the current fixation is masked; the window moves with each eye movement.[9][5] |
| Boundary | Rayner (1975)[10] | An invisible boundary is placed before a target word; while the reader fixates left of the boundary a preview (valid or manipulated) is shown, and it is replaced by the target word when the eyes cross the boundary.[9] |
| Moving mask | Rayner and Bertera (1979)[11] | A visual mask moving in synchrony with the eye covers foveal vision on each fixation; reading became difficult when foveal vision was masked.[11] |
Reingold and colleagues note that the gaze-contingent display methodology "has also had a tremendous influence in basic research on perception and cognition in areas such as reading and visual search".[2]
Simulation and computer graphics
According to Reingold and colleagues, flight simulation is the application area in which gaze-contingent multiresolutional displays have been used the longest and is still the most researched one, which they attribute to the large amount of funding available. They cite military flight simulators as an example of an application that needs a wraparound field of view, image resolution approaching the limit of the visual system and fast display updates at the same time.[2]
In computer graphics, Marc Levoy and Ross Whitaker presented gaze-directed volume rendering at the 1990 Symposium on Interactive 3D Graphics. Their ray tracer for volume data varied the number of rays cast per unit area of the image and the number of samples taken along each ray as functions of local retinal acuity, and they described an implementation using mip maps, an eye tracker and the Pixel-Planes 5 parallel raster display system; pending completion of Pixel-Planes 5 they tested the method with a simulator on a Stellar graphics supercomputer. Their preliminary results indicated that users were aware of the variable-resolution structure of the image but that the high-resolution "sweet spot" followed their gaze well.[12] By 2003, Reingold and colleagues listed several gaze-contingent multiresolutional VR display systems and reported that the published estimates of rendering-time savings were roughly 80% over a constant-resolution alternative; they also noted that most multiresolutional VR displays of the time were head-contingent rather than gaze-contingent.[2]
In 2012 Brian Guenter, Mark Finch, Steven Drucker, Desney Tan and John Snyder of Microsoft Research published "Foveated 3D Graphics" at SIGGRAPH Asia. Their system tracked the user's gaze and rendered three nested image layers around the gaze point at progressively lower sampling rates, speeding up rendering by a factor of 5 to 6 on a 1920x1080 desktop display and shading 10 to 15 times fewer pixels; the authors projected a 100-fold speedup for a 70 degree field of view at foveal resolution.[13]
Applications in VR and AR
Foveated rendering
Gaze-tracked foveated rendering is the most widespread use of gaze-contingent displays in VR; Duchowski calls the foveated gaze-contingent display "possibly the best example of passive use of gaze".[3] At SIGGRAPH Asia 2016, Anjul Patney and colleagues at NVIDIA reported that blurring the periphery reduces contrast and produces a sense of tunnel vision, and that with post-process contrast enhancement subjects tolerated up to twice as large a blur radius before detecting a difference from a non-foveated reference. They verified these findings on desktop displays and head-mounted displays fitted with high-speed gaze tracking, and their renderer reduced the number of shading operations by up to 70%.[14]
Consumer headsets with built-in eye tracking use the same principle. Sony listed foveated rendering among the visual features of the PlayStation VR2 in a January 2022 PlayStation Blog post that also described the headset's eye tracking.[15] For the Meta Quest Pro, UploadVR reported figures from Meta's own performance test app: eye-tracked foveated rendering saved 33% to 45% at default resolution and 36% to 52% at 1.5 times default resolution, compared with 26% to 36% and 34% to 43% for fixed foveated rendering, which does not follow the eyes.[16]
Gaze-contingent focus and depth cues
Because a conventional headset presents all content at one optical distance, several research systems use gaze to supply missing focus and depth cues. In a 2017 paper in PNAS, Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily Cooper and Gordon Wetzstein of described two prototype near-eye displays that used focus-tunable lenses, mechanically actuated displays and mobile gaze tracking to provide natural focus cues "by dynamically updating the system based on where a user looks in a virtual scene", and to correct common refractive errors.[6] Facebook Reality Labs' Half Dome varifocal prototype, shown at Facebook's F8 conference in 2018, combined moving display screens with eye tracking.[17]
Rendering-only approaches also exist. Gaze-contingent depth of field blurs scene content away from the fixated depth; Duchowski describes it as a software-based way to reduce the visual discomfort of stereoscopic displays caused by the vergence-accommodation conflict, alongside adjusting disparity locally at the 3D gaze location.[3] Robert Konrad, Anastasios Angelopoulos and Gordon Wetzstein introduced ocular parallax rendering, which uses gaze to reproduce the small depth-dependent image shifts on the retina that occur because the eye's centres of rotation and projection differ; their user studies found that the effect is clearly visible in most VR applications, provides an effective ordinal depth cue, and improves the impression of realistic depth.[18] Brooke Krajancich, Petr Kellnhofer and Wetzstein extended the idea to gaze-contingent stereo rendering, which models the gaze-dependent shift of the eye's no-parallax point; they reported improved disparity and shape distortion in VR and more consistent alignment of physical and rendered objects across depths in optical see-through AR.[19]
Gaze-contingent display hardware
Some prototypes move parts of the optics, not only the rendered image, with the eye. NVIDIA's Foveated AR display, presented at SIGGRAPH 2019, combined a traveling microdisplay relayed off a concave half-mirror for the high-resolution foveal region with a wide-field peripheral display: a projector-based Maxwellian-view display whose nodal point was moved to follow the pupil by a traveling holographic optical element. An infrared camera imaging the eye through the same optics drove both the foveal display position and the peripheral nodal point, and the focal depth of the foveal microdisplay could be varied. The authors showed prototypes with 30, 40 and 60 cycles per degree of foveal resolution over a net 85 by 78 degree field of view per eye.[20]
Redirected walking
Sun and colleagues used gaze contingency for locomotion rather than image quality. Their 2018 redirected walking method used a head- and eye-tracking VR headset to detect saccadic suppression and redirected the user during the resulting temporary blindness. They reported that saccades significantly increased the rotation gains available during redirection without introducing visual distortions or simulator sickness, which let the method work in small physical rooms with static or moving obstacles, including other users.[7]
Simulating visual field loss
Gaze-contingent displays can also degrade vision on purpose. Duchowski notes that they have been used to simulate arbitrary visual fields, including scotoma.[3] Francesca Fortenbaugh and colleagues extended Wilson Geisler and Jeffrey Perry's 2002 gaze-contingent multiresolution approach to 3D images transformed and rendered in real time, to study how central and peripheral vision contribute to navigation in virtual environments. In their test, seven participants walked to a target tree in a virtual forest while artificial head and eye-based delays were introduced; trial times increased under simulated peripheral field loss compared with a full field of view.[21]
Latency and perceptual limits
A central engineering problem is the delay between an eye movement and the display update. Duchowski describes eye tracking latency, which delays the appearance of the high-resolution inset, as "the greatest obstacle to practical utility of foveated rendering".[3] Reingold and colleagues identify two ways delays cause problems: if the update arrives late after a saccade, the eye may land on a degraded region; and an update that occurs well into a fixation (for example 70 ms or later) can be perceived as motion.[2]
Measured tolerances vary with the task and with what is measured:
| Study | Setting | Finding |
|---|---|---|
| Loschky and McConkie (2000), as summarized by Duchowski (2018) | Gaze-contingent multiresolutional display | To be indistinguishable from a full-resolution display, the high-resolution inset should appear within 7 ms of fixation onset; delays of about 15 ms were detectable but had minimal effect on visual task performance when the foveal inset radius was 4 degrees or more.[3] |
| Loschky and Wolverton (2007) | Gaze-contingent multiresolutional display | Image updates as late as 60 ms after an eye movement did not significantly increase the detectability of blur or motion transients; longer saccades led to more detection because they carried the eyes further into the low-resolution periphery.[22] |
| Albert, Patney, Luebke and Kim (2017) | Foveated rendering in VR | Added eye-tracking latency of 80 to 150 ms significantly reduced the acceptable amount of foveation; 20 to 40 ms did not, suggesting a total system latency of 50 to 70 ms could be tolerated. Larger foveal regions allowed more aggressive foveation.[23] |
Duchowski cautions that the 60 ms figure is measured from the end of the saccade and the 7 ms figure from the start of the fixation, so the two are not directly comparable.[3] Albert and colleagues framed their study around saccadic omission, which they suggested could allow the latency requirements to be relaxed.[23] One approach to hiding the remaining lag is to predict where a saccade will land while it is still in flight; Duchowski singles out saccade landing-position prediction by Elena Arabadzhiyska and colleagues (2017) as one of the most promising.[3]
Other open questions concern the shape of the high-resolution region, the choice between discrete and continuous resolution drop-off, and level-of-detail "popping", in which object details appear to pop in and out depending on their distance from the point of gaze.[2]
See also
References
- ↑ 1.0 1.1 Andrew T. Duchowski, Nathan Cournia, Hunter Murphy (2004-12). "Gaze-contingent displays: a review". CyberPsychology & Behavior, vol. 7, no. 6, pp. 621-634. https://doi.org/10.1089/cpb.2004.7.621. Retrieved 2026-10-15.
- ↑ 2.00 2.01 2.02 2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11 2.12 Eyal M. Reingold, Lester C. Loschky, George W. McConkie, David M. Stampe (2003). "Gaze-Contingent Multiresolutional Displays: An Integrative Review". Human Factors, vol. 45, no. 2, pp. 307-328. Human Factors and Ergonomics Society. doi:10.1518/hfes.45.2.307.27235. https://www.k-state.edu/psych/vcl/documents/Reingold_Loschky_McConkie_Stampe_2003_GCMRDs_An%20integrative%20review.pdf. Retrieved 2026-10-15.
- ↑ 3.00 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09 3.10 Andrew T. Duchowski (2018-06). "Gaze-based interaction: A 30 year retrospective". Computers & Graphics, vol. 73, pp. 59-69. https://doi.org/10.1016/j.cag.2018.04.002. Retrieved 2026-10-15.
- ↑ 4.0 4.1 4.2 George W. McConkie, Keith Rayner (1975-11). "The span of the effective stimulus during a fixation in reading". Perception & Psychophysics, vol. 17, no. 6, pp. 578-586. https://doi.org/10.3758/BF03203972. Retrieved 2026-10-15.
- ↑ 5.0 5.1 5.2 5.3 John M. Henderson, Fernanda Ferreira (1993-06). "Eye movement control during reading: Fixation measures reflect foveal but not parafoveal processing difficulty". Canadian Journal of Experimental Psychology, vol. 47, no. 2, pp. 201-221. https://doi.org/10.1037/h0078814. Retrieved 2026-10-15.
- ↑ 6.0 6.1 Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A. Cooper, Gordon Wetzstein (2017-02-13). "Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays". Proceedings of the National Academy of Sciences, vol. 114, no. 9, pp. 2183-2188. https://doi.org/10.1073/pnas.1617251114. Retrieved 2026-10-15.
- ↑ 7.0 7.1 Qi Sun, Anjul Patney, Li-Yi Wei, Omer Shapira, Jingwan Lu, Paul Asente, Suwen Zhu, Morgan McGuire, David Luebke, Arie Kaufman (2018-08-12). "Towards Virtual Reality Infinite Walking: Dynamic Saccadic Redirection". ACM Transactions on Graphics, vol. 37, no. 4 (Proceedings of SIGGRAPH 2018). NVIDIA Research. doi:10.1145/3197517.3201294. https://research.nvidia.com/publication/2018-08_towards-virtual-reality-infinite-walking-dynamic-saccadic-redirection. Retrieved 2026-10-15.
- ↑ Stephen M. Reder (1973-03). "On-line monitoring of eye-position signals in contingent and noncontingent paradigms". Behavior Research Methods & Instrumentation, vol. 5, no. 2, pp. 218-228. https://doi.org/10.3758/BF03200168. Retrieved 2026-10-15.
- ↑ 9.0 9.1 9.2 Christina Marx, Florian Hutzler, Sarah Schuster, Stefan Hawelka (2016-04-14). "On the Development of Parafoveal Preprocessing: Evidence from the Incremental Boundary Paradigm". Frontiers in Psychology, vol. 7, article 514. https://doi.org/10.3389/fpsyg.2016.00514. Retrieved 2026-10-15.
- ↑ Keith Rayner (1975-01). "The perceptual span and peripheral cues in reading". Cognitive Psychology, vol. 7, no. 1, pp. 65-81. https://doi.org/10.1016/0010-0285(75)90005-5. Retrieved 2026-10-15.
- ↑ 11.0 11.1 Keith Rayner, James H. Bertera (1979-10-26). "Reading Without a Fovea". Science, vol. 206, no. 4417, pp. 468-469. https://doi.org/10.1126/science.504987. Retrieved 2026-10-15.
- ↑ Marc Levoy, Ross Whitaker (1990-03). "Gaze-Directed Volume Rendering". Computer Graphics (Proc. 1990 Symposium on Interactive 3D Graphics), vol. 24, no. 2, pp. 217-223. Stanford Computer Graphics Laboratory. https://graphics.stanford.edu/papers/gaze-i3d90/. Retrieved 2026-10-15.
- ↑ Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, John Snyder (2012-11). "Foveated 3D Graphics". ACM Transactions on Graphics, vol. 31, no. 6 (SIGGRAPH Asia 2012). Microsoft Research. doi:10.1145/2366145.2366183. https://www.microsoft.com/en-us/research/publication/foveated-3d-graphics/. Retrieved 2026-10-15.
- ↑ Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, Aaron Lefohn (2016-12-05). "Towards Foveated Rendering for Gaze-Tracked Virtual Reality". ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2016). NVIDIA Research. doi:10.1145/2980179.2980246. https://research.nvidia.com/publication/2016-12_towards-foveated-rendering-gaze-tracked-virtual-reality. Retrieved 2026-10-15.
- ↑ Hideaki Nishino (2022-01-04). "PlayStation VR2 and PlayStation VR2 Sense controller: the next generation of VR gaming on PS5". PlayStation Blog. Sony Interactive Entertainment. https://blog.playstation.com/2022/01/04/playstation-vr2-and-playstation-vr2-sense-controller-the-next-generation-of-vr-gaming-on-ps5/. Retrieved 2026-10-15.
- ↑ David Heaney (2022-10-13). "Here's The Exact Performance Benefit Of Foveated Rendering On Quest Pro". UploadVR. https://www.uploadvr.com/quest-pro-foveated-rendering-performance/. Retrieved 2026-10-15.
- ↑ "Introducing the Team Behind Half Dome, Facebook Reality Labs' Varifocal Prototype". Meta. Meta Platforms. 2018-05-24. https://www.meta.com/blog/introducing-the-team-behind-half-dome-facebook-reality-labs-varifocal-prototype/. Retrieved 2026-10-15.
- ↑ Robert Konrad, Anastasios Angelopoulos, Gordon Wetzstein (2020-01-28). "Gaze-Contingent Ocular Parallax Rendering for Virtual Reality". ACM Transactions on Graphics, vol. 39, no. 2. https://doi.org/10.1145/3361330. Retrieved 2026-10-15.
- ↑ Brooke Krajancich, Petr Kellnhofer, Gordon Wetzstein (2020-11). "Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering". ACM Transactions on Graphics, vol. 39, no. 6. https://doi.org/10.1145/3414685.3417820. Retrieved 2026-10-15.
- ↑ Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, Peter Shirley, Josef Spjut, Morgan McGuire, David Luebke (2019-07-28). "Foveated AR: Dynamically-Foveated Augmented Reality Display". ACM Transactions on Graphics, vol. 38, no. 4 (SIGGRAPH 2019). NVIDIA Research. doi:10.1145/3306346.3322987. https://research.nvidia.com/publication/2019-07_foveated-ar-dynamically-foveated-augmented-reality-display. Retrieved 2026-10-15.
- ↑ Francesca C. Fortenbaugh, John C. Hicks, Lei Hao, Kathleen A. Turano (2007-08). "A technique for simulating visual field losses in virtual environments to study human navigation". Behavior Research Methods, vol. 39, no. 3, pp. 552-560. https://doi.org/10.3758/bf03193025. Retrieved 2026-10-15.
- ↑ Lester C. Loschky, Gary S. Wolverton (2007-12). "How Late Can You Update Gaze-Contingent Multiresolutional Displays Without Detection?". ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 3, no. 4, article 25. doi:10.1145/1314303.1314310. https://www.k-state.edu/psych/vcl/documents/Loschky_Wolverton_2007_How%20late%20can%20you%20update%20GCMRDs%20without%20detection.pdf. Retrieved 2026-10-15.
- ↑ 23.0 23.1 Rachel Albert, Anjul Patney, David Luebke, Joohwan Kim (2017-09-18). "Latency Requirements for Foveated Rendering in Virtual Reality". ACM Transactions on Applied Perception, vol. 14, no. 4, article 25. NVIDIA Research. doi:10.1145/3127589. https://research.nvidia.com/publication/2017-09_latency-requirements-foveated-rendering-virtual-reality. Retrieved 2026-10-15.