Jump to content

Eye tracking: Difference between revisions

From VR & AR Wiki
Newer edit →
VisualWikitext
Created page with "'''Eye tracking''' is a technology that detects and analyzes eye movements, gaze direction, and related metrics. In the context of virtual reality (VR) and augmented reality (AR), eye tracking enables headsets to determine precisely where users are looking, creating more immersive and efficient experiences. This technology has become increasingly important in modern head-mounted displays (HMDs), enabling advanced features like foveated rendering,..."
 
No edit summary
Line 1: Line 1:
'''Eye tracking''' is a technology that detects and analyzes [[eye movement]]s, [[gaze]] direction, and related metrics. In the context of [[virtual reality]] (VR) and [[augmented reality]] (AR), eye tracking enables headsets to determine precisely where users are looking, creating more immersive and efficient experiences. This technology has become increasingly important in modern [[head-mounted display]]s (HMDs), enabling advanced features like [[foveated rendering]], [[gaze-based interaction]], and enhanced user analytics.<ref>Duchowski, A. T. (2017). Eye Tracking Methodology: Theory and Practice. Springer.</ref>
# Eye Tracking
 
'''Eye tracking''' is a sensor technology that measures [[eye]] positions and [[eye movement]]. In the context of [[virtual reality]] (VR) and [[augmented reality]] (AR), it refers to the integration of sensors within [[VR headset]]s or [[AR headset]]s to determine precisely where the user is looking ([[gaze]]) in real-time within the [[virtual environment]] or overlaid digital interface. By accurately monitoring characteristics like pupil position, corneal reflections, and eye movements such as [[saccades]] and fixations, this technology enables more immersive, efficient, and intuitive user experiences. It has become an increasingly critical feature in modern [[head-mounted display]]s (HMDs), driving advancements in rendering, interaction, analytics, and social presence.<ref name="Duchowski2017">Duchowski, A. T. (2017). *Eye Tracking Methodology: Theory and Practice*. Springer.</ref>
 
== History ==
While its integration into consumer VR/AR is relatively recent, the systematic study of eye movements dates back to the 19th century. [[Louis Émile Javal]] noted in 1879 that reading involved discrete stops (fixations) and rapid movements (saccades).<ref name="Rayner1998">Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. *Psychological Bulletin, 124*(3), 372–422.</ref> Early tracking devices included [[Edmund Huey]]'s contact lens-based tracker (~1908)<ref name="Huey1908">Huey, E.B. (1908). *The Psychology and Pedagogy of Reading*. Macmillan.</ref> and [[Guy Thomas Buswell]]'s film-based systems in the 1930s. [[Alfred L. Yarbus]]'s work in the 1960s highlighted how viewing patterns are task-dependent.<ref name="Yarbus1967">Yarbus, A.L. (1967). *Eye Movements and Vision*. Plenum Press.</ref> These foundational efforts paved the way for modern video-based and integrated tracking systems.


== Technical Principles ==
== Technical Principles ==


=== Tracking Methods ===
=== Tracking Methods ===
Most eye tracking systems in contemporary VR and AR headsets employ one of several core technologies:


Most eye tracking systems in VR and AR employ one of several core technologies:
*   '''[[Pupil Center Corneal Reflection]] (PCCR)''': This is the predominant method. It involves:<ref name="Holmqvist2011">Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). *Eye tracking: A comprehensive guide to methods and measures*. Oxford University Press.</ref><ref name="RoadToVREyeTracking">Lang, B. (2021, July 21). *Casual Explainer: What is Eye-tracking & How Does it Work?* Road to VR. https://www.roadtovr.com/casual-explainer-what-is-eye-tracking-how-does-it-work/</ref>
 
    *   '''Illumination:''' [[Infrared]] (IR) light-emitting diodes ([[LED]]s) safely illuminate the eye. IR light is used because it is invisible to the human eye, preventing distraction, and provides high contrast for [[camera]]s.
* '''Pupil Center Corneal Reflection (PCCR)''' - The most common method in modern headsets, which uses [[infrared light]] to create reflections on the cornea and tracks these reflections relative to the pupil center.<ref>Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press.</ref>
    *  '''Imaging:''' Small, high-[[frame rate]] infrared cameras capture images of the eye, specifically tracking the center of the [[pupil]] and the reflection(s) of the IR light off the surface of the [[cornea]] (known as glints).
 
    *   '''[[Algorithm|Algorithmic Processing]]:''' Sophisticated [[computer vision]] and [[image processing]] algorithms analyze the captured images. By calculating the vector between the pupil center and the corneal reflection(s), the system determines the eye's orientation and calculates the user's gaze point with high accuracy.
* '''Video-based eye tracking''' - Uses small cameras aimed at the eyes to capture images that are then analyzed with [[computer vision]] algorithms to determine gaze direction.<ref>Hansen, D. W., & Ji, Q. (2010). In the eye of the beholder: A survey of models for eyes and gaze. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3), 478-500.</ref>
    *  '''[[Calibration]]:''' A per-user calibration process is usually required upon first use, and sometimes periodically, to account for individual differences in eye physiology (e.g., corneal shape, pupil size range) and the precise fit of the headset. This typically involves the user looking at specific points displayed within the headset.
 
* '''Electrooculography (EOG)''' - Measures the electrical potential between electrodes placed around the eye, which changes as the eye moves. Less common in VR/AR but useful in some specialized applications.<ref>Bulling, A., Ward, J. A., Gellersen, H., & Tröster, G. (2011). Eye movement analysis for activity recognition using electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(4), 741-753.</ref>
 
=== Key Components ===


A typical eye tracking system in VR/AR headsets consists of:
*  '''Video-based eye tracking (Shape/Feature Tracking)''': Uses cameras aimed at the eyes to capture images, which are then analyzed using computer vision algorithms to identify eye features (pupil outline, iris texture, blood vessels) to determine gaze direction without necessarily relying on corneal reflections.<ref name="Hansen2010">Hansen, D. W., & Ji, Q. (2010). In the eye of the beholder: A survey of models for eyes and gaze. *IEEE Transactions on Pattern Analysis and Machine Intelligence, 32*(3), 478-500.</ref> PCCR is often considered a subset of this broader category.


* '''Illuminators''' - Usually infrared LEDs that provide consistent lighting without distracting the user
*   '''[[Electrooculography]] (EOG)''': Measures the electrical potential difference between electrodes placed on the skin around the eyes. This potential changes predictably as the eye rotates. While less common for high-accuracy gaze pointing in consumer VR/AR due to lower precision and susceptibility to muscle noise, it can be used for detecting larger eye movements or in specialized applications.<ref name="Bulling2011">Bulling, A., Ward, J. A., Gellersen, H., & Tröster, G. (2011). Eye movement analysis for activity recognition using electrooculography. *IEEE Transactions on Pattern Analysis and Machine Intelligence, 33*(4), 741-753.</ref>
* '''Cameras''' - Specialized infrared cameras that capture eye images
* '''Processing algorithms''' - Software that analyzes the captured images to determine eye position and movement
* '''Calibration system''' - Process to adjust the system to individual users' eye characteristics<ref>Kar, A., & Corcoran, P. (2017). A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms. IEEE Access, 5, 16495-16519.</ref>


=== Eye Movement Types ===
*  '''[[Scleral search coil]]''': Involves wearing a special contact lens containing a wire coil. The user sits within a magnetic field, and eye movements induce currents in the coil, providing extremely precise measurements. This is highly invasive and primarily used in laboratory research, not consumer VR/AR.<ref name="Robinson1963">Robinson, D.A. (1963). A method of measuring eye movement using a scleral search coil in a magnetic field. *IEEE Transactions on Biomedical Engineering, BME-10*(4), 137–145.</ref>


Eye tracking systems in VR/AR can detect several types of eye movements:
=== Key Components (PCCR-based) ===
A typical PCCR eye tracking system integrated into a VR/AR headset consists of:
*  '''Illuminators''': Infrared LEDs providing consistent, non-visible lighting.
*  '''Cameras''': Specialized high-speed infrared cameras positioned to capture clear images of both eyes.
*  '''Processing Unit''': Either onboard [[System on a Chip|SoC]] resources or dedicated hardware to run the detection and gaze calculation algorithms in real-time.
*  '''Calibration Software''': [[Software]] routines guiding the user through calibration and storing individual profiles.<ref name="Kar2017">Kar, A., & Corcoran, P. (2017). A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms. *IEEE Access, 5*, 16495-16519.</ref>


* '''[[Saccade]]s''' - Rapid movements between fixation points (30-80 ms)
=== Eye Movement Types Measured ===
* '''Fixations''' - Relatively stable gazes on a specific area (200-300 ms)
Eye tracking systems in VR/AR can detect and analyze various types of eye movements and states:
* '''Smooth pursuits''' - Movements that track moving objects
*   '''[[Fixations]]''': Periods when the gaze remains relatively stable on a specific area (typically > 100-200 ms), indicating visual attention.
* '''Vergence''' - Movements where eyes move in opposite directions to focus on objects at different depths
*   '''[[Saccades]]''': Rapid, ballistic movements shifting the gaze between fixation points (typically 30-120 ms).
* '''Pupil dilation/constriction''' - Changes in pupil size which can indicate cognitive load or emotional response<ref>Leigh, R. J., & Zee, D. S. (2015). The neurology of eye movements. Oxford University Press.</ref>
*   '''[[Smooth pursuit]]''': Movements allowing the eyes to smoothly follow a moving object.
*   '''[[Vergence]]''': The simultaneous movement of both eyes in opposite directions to obtain or maintain single binocular vision, critical for focusing on objects at different depths.
*   '''[[Pupil dilation|Pupil Size / Dilation]]''': Changes in pupil diameter (pupillometry), which can correlate with changes in light levels, cognitive load, emotional arousal, or interest.<ref name="Leigh2015">Leigh, R. J., & Zee, D. S. (2015). *The neurology of eye movements*. Oxford University Press.</ref>
*  '''Blinks''': Detection of eyelid closures.


== Applications in VR and AR ==
== Applications in VR and AR ==
Eye tracking unlocks numerous capabilities enhancing VR and AR experiences:


=== Foveated Rendering ===
*  '''[[Foveated rendering]]''': Arguably the most impactful application for performance. By knowing precisely where the user is looking, the system renders the scene at maximum [[resolution]] only in the small, central area of the user's gaze (corresponding to the eye's high-acuity fovea), while rendering the peripheral areas at progressively lower resolutions. This mimics human vision and can drastically reduce the computational load on the [[graphics processing unit]] (GPU) – potentially by 30% to over 70% – allowing for higher fidelity graphics, increased frame rates, reduced [[latency]], or lower power consumption, all without a noticeable loss in perceived visual quality.<ref name="Patney2016">Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C., Benty, N., Luebke, D., & Lefohn, A. (2016). Towards foveated rendering for gaze-tracked virtual reality. *ACM Transactions on Graphics, 35*(6), 179.</ref><ref name="TobiiFoveated">[https://www.tobii.com/blog/eye-tracking-in-vr-a-vital-component Tobii Blog: Eye Tracking in VR — A Vital Component]. Retrieved Nov 17, 2023.</ref><ref name="NvidiaFoveated">NVIDIA Corporation. (n.d.). *Maximize VR Performance with Foveated Rendering*. NVIDIA Developer. Retrieved November 16, 2023, from https://developer.nvidia.com/vrworks/graphics/foveatedrendering</ref>


One of the most significant applications of eye tracking in VR/AR is [[foveated rendering]], a technique that renders images at full resolution only where the user is looking, while reducing detail in peripheral vision. This mimics the human visual system's natural function and can significantly reduce computational requirements.<ref>Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C., Benty, N., Luebke, D., & Lefohn, A. (2016). Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics, 35(6), 179.</ref>
*  '''Natural Interaction / Gaze-Based Interaction''': Eye tracking enables more intuitive control schemes:
    *  '''Gaze Selection/Pointing''': Allows users to select objects, menu items, or targets simply by looking at them. This is often combined with a confirmation action like a button press on a [[controller (computing)|controller]], a [[hand tracking]] gesture (e.g., pinch), or a short dwell time.<ref name="Piumsomboon2017">Piumsomboon, T., Lee, G., Lindeman, R. W., & Billinghurst, M. (2017). Exploring natural eye-gaze-based interaction for immersive virtual reality. *IEEE Symposium on 3D User Interfaces*, 36-39.</ref>
    *  '''Intent Prediction''': Systems can anticipate user actions or needs based on gaze patterns (e.g., highlighting an object the user looks at intently).
    *  '''Gaze-directed Locomotion''': Steering movement within the virtual world based on gaze direction.


Benefits include:
*  '''Enhanced Social Presence / [[Avatar]] Realism''': Realistic eye movements, including subtle saccades, blinks, and responsive gaze shifts, can be mirrored onto a user's avatar in social VR applications. This significantly enhances [[non-verbal communication]] and the feeling of [[social presence]] and connection when interacting with others.<ref name="MetaAvatarsEyeTracking">Meta Platforms, Inc. (2022, October 11). *Meta Quest Pro: A New Way to Work, Create and Collaborate*. Meta Quest Blog. https://www.meta.com/blog/quest/meta-quest-pro-vr-headset-features-price-release-date/</ref>
* 30-60% reduction in GPU processing requirements
* Increased frame rates and reduced latency
* Ability to render more complex scenes
* Extended battery life in standalone headsets


=== Natural Interaction ===
*  '''User Analytics and Research''': Eye tracking provides invaluable objective data for:
    *  '''[[Usability testing]] & User Experience (UX) Research''': Understanding how users visually explore and interact with interfaces or environments.
    *  '''[[Attention mapping]]''': Creating heatmaps and gaze plots to visualize areas of interest and attention duration.
    *  '''Cognitive Load Assessment''': Measuring mental workload through metrics like pupil dilation, blink rate, and fixation patterns.<ref name="Clay2019">Clay, V., König, P., & König, S. (2019). Eye tracking in virtual reality. *Journal of Eye Movement Research, 12*(1).</ref>
    *  '''Training and Simulation Analysis''': Assessing trainee attention, situational awareness, and decision-making processes in professional simulations (e.g., medical, aviation).


Eye tracking enables more intuitive ways to interact with virtual environments:
*  '''Automatic [[Interpupillary distance|IPD]] Adjustment''': Some headsets utilize the eye tracking cameras to automatically measure the user's interpupillary distance (the distance between the centers of the pupils) and mechanically adjust the lens spacing for optimal visual clarity, stereo depth perception, and user comfort.


* '''Gaze selection''' - Allows users to select objects simply by looking at them
*   '''[[Accessibility]]''': Eye tracking offers a powerful hands-free input modality for users with limited physical mobility, enabling them to navigate interfaces, communicate (e.g., gaze typing), and control applications within VR/AR.<ref name="Yuan2020">Yuan, Z., Bi, T., Muntean, G. M., & Ghinea, G. (2020). Perceived synchronization of mulsemedia services. *IEEE Transactions on Multimedia, 17*(7), 957-966. Note: While this ref discusses synchronization, accessibility is a widely cited benefit of eye tracking.</ref>
* '''Intent prediction''' - Systems can anticipate user actions based on gaze patterns
* '''Social VR''' - Enables realistic [[avatar]] eye movements in virtual social interactions, greatly enhancing presence<ref>Piumsomboon, T., Lee, G., Lindeman, R. W., & Billinghurst, M. (2017). Exploring natural eye-gaze-based interaction for immersive virtual reality. IEEE Symposium on 3D User Interfaces, 36-39.</ref>


=== User Analytics and Research ===
*  '''Adaptive Optics / [[Varifocal display]]s''': Eye tracking is essential for dynamic varifocal displays, which adjust their focal plane based on where the user is looking in virtual depth. This helps address the [[vergence-accommodation conflict]], potentially reducing eye strain and improving visual realism.<ref name="Akeley2004">Akeley, K., Watt, S.J., Girshick, A.R., & Banks, M.S. (2004). A stereo display prototype with multiple focal distances. *ACM Transactions on Graphics, 23*(3), 804–813.</ref>


Eye tracking provides valuable data for:
*  '''[[Dynamic Distortion Compensation]]''': Real-time adjustments to lens distortion correction based on precise eye position relative to the lens center can improve perceived sharpness across the field of view.<ref name="TobiiFoveated"/>


* '''User experience research''' - Understanding how users interact with virtual interfaces
== Current Implementations ==
* '''[[Attention mapping]]''' - Creating heatmaps of where users focus in virtual environments
* '''Cognitive load assessment''' - Measuring mental workload through pupil dilation and blink patterns
* '''Training and simulation''' - Analyzing trainee attention patterns in professional simulations<ref>Clay, V., König, P., & König, S. (2019). Eye tracking in virtual reality. Journal of Eye Movement Research, 12(1).</ref>
 
=== Accessibility Features ===
 
Eye tracking enables VR/AR experiences for users with limited mobility:
 
* Hands-free navigation and control
* Assistive communication through gaze typing
* Customized interfaces based on individual capabilities<ref>Yuan, Z., Bi, T., Muntean, G. M., & Ghinea, G. (2020). Perceived synchronization of mulsemedia services. IEEE Transactions on Multimedia, 17(7), 957-966.</ref>


== Current Implementations ==
Several commercially available VR and AR headsets incorporate integrated eye tracking:


=== VR Headsets with Eye Tracking ===
=== VR Headsets with Eye Tracking ===
 
*   '''[[Apple Vision Pro]]''': Uses high-precision eye tracking as a primary input method (combined with hand gestures and voice), enabling UI navigation, selection, and foveated rendering.<ref name="AppleVisionPro">Apple Inc. (n.d.). *Apple Vision Pro*. Apple. Retrieved November 16, 2023, from https://www.apple.com/apple-vision-pro/</ref>
Several commercial VR headsets now incorporate eye tracking:
*  '''[[Meta Quest Pro]]''': Features inward-facing sensors for eye and face tracking, primarily used for foveated rendering and driving realistic avatar expressions in social applications.<ref name="MetaAvatarsEyeTracking"/>
 
*  '''[[PlayStation VR2]]''': Integrates [[Tobii]] eye tracking technology for foveated rendering, gaze-based interactions in games, and enhanced immersion.<ref name="SonyPSVR2">Sony Interactive Entertainment. (2023). *PS VR2 Features*. PlayStation.com. https://www.playstation.com/en-us/ps-vr2/features/</ref>
* '''[[Apple Vision Pro]]''' - Uses high-precision eye tracking as a primary input method alongside hand tracking and voice commands. Features dual micro-OLED displays and includes eye tracking for both navigation and [[foveated rendering]].<ref>Apple Inc. (2023). Apple Vision Pro Technical Specifications. Apple.com.</ref>
*   '''[[HTC VIVE Pro Eye]]''': An earlier enterprise-focused headset integrating Tobii eye tracking (accuracy ~0.5-1.1 degrees, 120Hz). Newer VIVE models may offer eye tracking via add-ons (e.g., VIVE Focus 3 Eye Tracker).<ref name="ViveProEye">HTC Corporation. (n.d.). *VIVE Pro Eye*. VIVE. Retrieved November 16, 2023, from https://www.vive.com/us/product/vive-pro-eye/overview/</ref>
 
*   '''[[Varjo]] XR-4, XR-3, VR-3, Aero''': High-end professional headsets featuring industrial-grade eye tracking (sub-degree accuracy, 200Hz) for demanding simulation, research, and design applications.<ref name="VarjoAero">Varjo Technologies Oy. (n.d.). *Varjo Aero*. Varjo. Retrieved November 16, 2023, from https://varjo.com/products/aero/</ref><ref name="VarjoXR3">Varjo Technologies. (2021). *Varjo XR-3 Technical Specifications*. Varjo.com.</ref>
* '''[[HTC VIVE Pro Eye]]''' - Integrates Tobii eye tracking technology with accuracy of 0.5-1.1 degrees and a tracking frequency of 120Hz. Supports foveated rendering and gaze-based user interface interaction.<ref>HTC Corporation. (2019). VIVE Pro Eye User Guide. Vive.com.</ref>
*  '''[[Pimax]] Crystal''': Consumer-focused high-resolution headset incorporating eye tracking for features like foveated rendering and automatic IPD adjustment.<ref name="PimaxCrystal">Pimax Technology (Shanghai) Co., Ltd. (n.d.). *Pimax Crystal*. Pimax. Retrieved November 16, 2023, from https://pimax.com/crystal/</ref>
 
*   '''[[Pico 4 Enterprise]]''' (previously Pico Neo 3 Pro Eye): Enterprise headsets integrating Tobii eye tracking for business applications.<ref name="PicoNeo3Eye">Pico Interactive. (2021). *Pico Neo 3 Pro Eye Specifications*. Pico-interactive.com.</ref>
* '''[[Varjo VR-3]]''' and '''[[Varjo XR-3]]''' - Feature industrial-grade eye tracking with sub-degree accuracy and a 200Hz tracking rate. Used primarily for professional applications such as training, simulation, and research.<ref>Varjo Technologies. (2021). Varjo VR-3 Technical Specifications. Varjo.com.</ref>
*   '''[[HP Reverb G2 Omnicept Edition]]''': Featured integrated eye tracking (along with other biometric sensors) for enterprise use cases.<ref name="HPOmnicept">HP Development Company, L.P. (n.d.). *HP Reverb G2 Omnicept Edition VR Headset*. HP.com. Retrieved November 16, 2023, from https://www.hp.com/us-en/vr/reverb-g2-vr-headset-omnicept-edition.html</ref>
 
* '''[[Pico Neo 3 Pro Eye]]''' - Incorporates Tobii eye tracking for enterprise applications with 90Hz refresh rate and 6DoF tracking.<ref>Pico Interactive. (2021). Pico Neo 3 Pro Eye Specifications. Pico-interactive.com.</ref>
 
* '''[[Meta Quest Pro]]''' - Features internal and external sensors for face and eye tracking to facilitate more realistic avatars and social interactions.<ref>Meta. (2022). Meta Quest Pro Features and Specifications. Meta.com.</ref>


=== AR Headsets with Eye Tracking ===
=== AR Headsets with Eye Tracking ===
 
Eye tracking is also crucial for interaction and performance in AR:
Eye tracking is equally important in AR implementations:
*   '''[[Microsoft HoloLens 2]]''': Uses eye tracking for user calibration, interaction (gaze targeting), and potentially performance optimization. Reported accuracy around 1.5 degrees.<ref name="HoloLens2">Microsoft Corporation. (2019). *HoloLens 2 Hardware Details*. Microsoft.com.</ref>
 
*   '''[[Magic Leap 2]]''': Incorporates eye tracking for input (gaze, dwell), foveated rendering (on segmented dimmer), user calibration, and analytics.<ref name="MagicLeap2">Magic Leap, Inc. (2022). *Magic Leap 2 Technical Overview*. MagicLeap.com.</ref>
* '''[[Microsoft HoloLens 2]]''' - Uses eye tracking for improved user interface interaction and application control with reported accuracy of about 1.5 degrees.<ref>Microsoft Corporation. (2019). HoloLens 2 Hardware Details. Microsoft.com.</ref>
 
* '''[[Magic Leap 2]]''' - Incorporates eye tracking for interface control and developer analytics with a reported field of view of 70° diagonal.<ref>Magic Leap, Inc. (2022). Magic Leap 2 Technical Overview. MagicLeap.com.</ref>
 
* '''[[Nreal Light]]''' - Features basic eye tracking capabilities for user interface interactions.<ref>Nreal. (2020). Nreal Light Technical Specifications. Nreal.io.</ref>


== Technical Specifications and Performance Metrics ==
== Technical Specifications and Performance Metrics ==


=== Key Performance Indicators ===
=== Key Performance Indicators ===
 
The quality and usability of eye tracking systems are measured using several critical metrics:
The performance of eye tracking systems is measured using several critical metrics:
*   '''Accuracy''': The average difference between the true gaze point and the system's reported gaze point, typically measured in degrees of visual angle. Good systems achieve 0.5° to 1.5° accuracy within the central field of view.
 
*   '''Precision''': The consistency or reproducibility of measurements for the same gaze point (RMS of successive samples). Often between 0.1° and 0.5°.
* '''Accuracy''' - Typically measured in degrees of visual angle, with industry standards ranging from 0.5° to 1.5°
*   '''Sampling Rate''': The frequency at which eye position is measured, expressed in Hertz (Hz). Consumer systems range from 30Hz to 120Hz or more, while research systems can exceed 1000Hz. Higher rates capture more detail about rapid movements like saccades.
* '''Precision''' - The consistency of measurements, usually between 0.1° and 0.5°
*   '''Latency''': The time delay between an actual eye movement and when it is detected and reported by the system. Crucial for real-time applications like foveated rendering and interaction, ideally below 20ms, though system latency can sometimes be higher (e.g., 45-81 ms reported in some studies).<ref name="SpringerReview2022">Mack, S., et al. (2022). A survey on eye tracking in virtual and augmented reality. *Virtual Reality*, 27, 1597–1625. https://link.springer.com/article/10.1007/s10055-022-00738-z</ref>
* '''Sampling rate''' - Frequency of eye position measurement, ranging from 30Hz in basic systems to 250Hz or higher in research-grade equipment
*   '''Robustness / Tracking Ratio''': The percentage of time the system successfully tracks the eyes under various conditions (e.g., different users, lighting, eyewear).
* '''Latency''' - Time delay between eye movement and system detection, ideally below 20ms for VR/AR applications
*  '''Field of View / Freedom of Head Movement''': The range of eye rotation and head position within which the system maintains tracking.
* '''Robustness''' - Performance across different users, lighting conditions, and use scenarios<ref>Blignaut, P. (2018). Using eye tracking to assess user experience: A case of a mobile banking application. In ACM International Conference Proceeding Series, 219-228.</ref>
<ref name="Blignaut2018">Blignaut, P. (2018). Using eye tracking to assess user experience: A case of a mobile banking application. In *ACM International Conference Proceeding Series*, 219-228.</ref>


=== Calibration Methods ===
=== Calibration Methods ===
 
Achieving specified accuracy typically requires individual user calibration:
Most eye tracking systems require calibration to achieve optimal performance:
*   '''Point Calibration''': The most common method. The user looks sequentially at several points displayed on the screen while the system records corresponding eye data to build a mapping model.
 
*   '''Smooth Pursuit Calibration''': The user follows one or more moving targets across the screen.
* '''Point calibration''' - User looks at specific points on screen while the system measures eye positions
*   '''Implicit/Adjustment Calibration''': Systems that attempt to calibrate or refine calibration based on natural viewing behavior during normal use, potentially reducing user friction.
* '''Pursuit calibration''' - User follows moving targets
*  '''Calibration Drift''': Accuracy can degrade over time due to headset slippage or physiological changes, potentially requiring recalibration. Some studies note significant drift (e.g., 30% accuracy loss) within minutes under certain conditions.<ref name="SpringerReview2022"/>
* '''Implicit calibration''' - System calibrates through normal use without specific user actions<ref>Santini, T., Fuhl, W., & Kasneci, E. (2018). CalibMe: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction. CHI Conference on Human Factors in Computing Systems, 1-6.</ref>
<ref name="Santini2018">Santini, T., Fuhl, W., & Kasneci, E. (2018). CalibMe: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction. *CHI Conference on Human Factors in Computing Systems*, 1-6.</ref>


== Challenges and Limitations ==
== Challenges and Limitations ==


Despite significant advances, eye tracking in VR/AR faces several challenges:
Despite significant progress, eye tracking in VR/AR still faces hurdles:


=== Technical Challenges ===
=== Technical Challenges ===
 
*   '''Individual Variations''': Differences in eye physiology (e.g., corneal curvature, pupil size, eyelid shape, ethnicity-related features, "droopy" eyelids) can impact tracking accuracy and robustness.
* '''Individual variations''' - Eye physiology differs significantly between users, affecting tracking accuracy
*   '''Eyewear Compatibility''': Prescription glasses (especially with thick lenses, bifocals, or certain coatings) and some types of contact lenses can interfere with IR illumination and camera imaging, degrading performance.
* '''Eyewear compatibility''' - Glasses and contact lenses can interfere with tracking systems
*   '''Processing Requirements''': Real-time, high-frequency eye tracking requires significant computational resources, impacting overall system performance and battery life, especially on standalone mobile headsets.
* '''Processing requirements''' - High-frequency eye tracking requires substantial computational resources
*   '''Power Consumption''': The cameras and illuminators continuously consume power, contributing to battery drain in untethered devices.
* '''Power consumption''' - A concern particularly for standalone and mobile devices<ref>Majaranta, P., & Bulling, A. (2014). Eye tracking and eye-based human-computer interaction. In Advances in physiological computing, 39-65. Springer.</ref>
*  '''Accuracy/Latency Trade-offs''': Achieving both high accuracy and very low latency simultaneously remains challenging.
<ref name="Majaranta2014">Majaranta, P., & Bulling, A. (2014). Eye tracking and eye-based human-computer interaction. In *Advances in physiological computing*, 39-65. Springer.</ref>


=== User Experience Concerns ===
=== User Experience Concerns ===
 
*   '''Calibration Process''': The need for calibration can be perceived as inconvenient, especially if required frequently.
* '''Calibration fatigue''' - Frequent recalibration can frustrate users
*   '''[[Privacy]] Implications''': Eye tracking data is highly sensitive [[biometric data]]. It can potentially reveal information about identity, attention focus, cognitive state, emotional responses, interests, and even certain health conditions (e.g., fatigue, intoxication, neurological disorders), raising significant privacy concerns if collected, stored, or shared improperly.<ref name="Kroger2020">Kröger, J. L., Lutz, O. H. M., & Müller, F. (2020). What does your gaze reveal about you? On the privacy implications of eye tracking. *Privacy and Identity Management*, 226-241.</ref><ref name="EFFPrivacyVRAR">Crockford, K., & Electronic Frontier Foundation. (2020, November 19). *The Privacy Bird Isn't Real: Your VR/AR Data Is*. Electronic Frontier Foundation. https://www.eff.org/deeplinks/2020/11/privacy-bird-isnt-real-your-vrar-data</ref>
* '''Privacy implications''' - Eye tracking data can reveal significant personal information
*  '''The "[[Uncanny Valley]]" Effect''': Imperfectly synchronized or unnatural avatar eye movements can appear disturbing rather than enhance social presence.
* '''The "Uncanny Valley" effect''' - If avatar eye movements aren't perfectly synchronized, they can appear disturbing<ref>Vrij, A., & Mann, S. (2020). Eye movements as a detection tool: a review and theoretical framework. Frontiers in Psychology, 11, 1538.</ref>


=== Accessibility Issues ===
=== Accessibility Issues ===
*  '''Compatibility with Eye Conditions''': Certain ophthalmological conditions (e.g., strabismus, nystagmus, ptosis, corneal scarring) can significantly impair or prevent accurate eye tracking for affected individuals.
*  '''Make-up Interference''': Certain types of mascara or eyeliner can sometimes interfere with tracking.


* '''Compatibility with eye conditions''' - Users with strabismus, nystagmus, or other eye conditions may experience reduced tracking quality
<ref name="Titz2018">Titz, J., Scholz, A., & Sedlmeier, P. (2018). Comparing eye trackers by correlating their eye-metric data. *Behavior Research Methods, 50*(5), 1853-1863.</ref>
* '''Cultural differences''' in eye movement patterns
* '''Age-related variations''' in pupil responsiveness and eye movement<ref>Titz, J., Scholz, A., & Sedlmeier, P. (2018). Comparing eye trackers by correlating their eye-metric data. Behavior Research Methods, 50(5), 1853-1863.</ref>


== Future Developments ==
== Future Developments ==


The field of eye tracking in VR/AR continues to advance rapidly:
The field continues to evolve rapidly:


=== Emerging Technologies ===
=== Emerging Technologies ===
 
*   '''Lower Power, Smaller Sensors''': Miniaturization and improved power efficiency for better integration into lighter HMDs.
* '''Micro LED-based trackers''' - Smaller, more power-efficient tracking systems
*   '''[[Artificial intelligence|AI]]/[[Machine learning|ML]]-Enhanced Tracking''': Using neural networks to improve robustness across diverse users, reduce calibration needs, and potentially infer more complex states from eye data.<ref name="Fuhl2016">Fuhl, W., Santini, T., Kasneci, G., & Kasneci, E. (2016). PupilNet: Convolutional neural networks for robust pupil detection. *CoRR, abs/1601.04902*.</ref>
* '''Neural network approaches''' - AI-enhanced tracking that adapts to individual users
*  '''Advanced Optical Designs''': Novel lens and camera configurations to improve tracking quality, especially for users with eyewear.
* '''Multispectral imaging''' - Using multiple light wavelengths for improved accuracy
*  '''Sensor Fusion''': Combining eye tracking data with other sensor inputs (e.g., [[electroencephalography]] (EEG), head tracking, facial expression tracking) for richer interaction and analysis.
* '''Non-visible light tracking''' - Advanced techniques that don't require infrared illumination<ref>Fuhl, W., Santini, T., Kasneci, G., & Kasneci, E. (2016). PupilNet: Convolutional neural networks for robust pupil detection. CoRR, abs/1601.04902.</ref>


=== Research Directions ===
=== Research Directions ===
 
*   '''Predictive Tracking''': Algorithms that anticipate eye movements to compensate for system latency, enabling smoother foveated rendering and interaction.
* '''Combined eye-brain interfaces''' - Integrating eye tracking with [[electroencephalography]] (EEG) for enhanced interaction
*   '''Emotion and Cognitive State Recognition''': Refining models to reliably infer affective states or cognitive load from eye metrics for adaptive interfaces or mental health applications.
* '''Emotion detection''' - Using pupil dilation and eye movement patterns to infer emotional states
*   '''Standardization''': Developing common metrics, APIs, and data formats to facilitate cross-platform development and research comparison.
* '''Predictive tracking''' - Algorithms that anticipate eye movements to reduce perceived latency
*   '''Longitudinal Tracking''': Understanding how gaze patterns change over extended use and adapting systems accordingly.
* '''Cross-platform standardization''' - Efforts to create universal eye tracking metrics and APIs<ref>Duchowski, A. T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M., & Giannopoulos, I. (2018). The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation. CHI Conference on Human Factors in Computing Systems, 1-13.</ref>
<ref name="Duchowski2018">Duchowski, A. T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M., & Giannopoulos, I. (2018). The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation. *CHI Conference on Human Factors in Computing Systems*, 1-13.</ref>


== Software Development and APIs ==
== Software Development and APIs ==


=== Development Frameworks ===
Integrating eye tracking into applications requires specific software support:
 
Several platforms offer tools for eye tracking integration:
 
* '''[[Unity XR Interaction Toolkit]]''' - Provides eye tracking input support for Unity developers
* '''[[Unreal Engine Eye Tracking Interface]]''' - API for implementing eye tracking in Unreal Engine applications
* '''[[OpenXR]]''' - Offers eye tracking extension for cross-platform development
* '''[[Tobii XR SDK]]''' - Specialized development kit for Tobii eye tracking hardware<ref>Tobii Technology. (2022). Tobii XR SDK Documentation. Tobii.com.</ref>
 
=== Data Processing Approaches ===


Developers can access eye tracking data at various levels:
### Development Frameworks and APIs
*  '''[[Unity (game engine)|Unity]]''': Provides APIs through its XR Interaction Toolkit and potential vendor-specific SDKs (e.g., Tobii XR SDK).
*  '''[[Unreal Engine]]''': Offers native interfaces and plugins for accessing eye tracking data.
*  '''[[OpenXR]]''': The cross-platform standard includes an `XR_EXT_eye_gaze_interaction` extension, allowing developers to write more portable code.
*  '''Vendor SDKs''': Companies like [[Tobii]] provide dedicated Software Development Kits offering fine-grained control and optimized features for their hardware.<ref name="TobiiSDK">Tobii Technology. (2023). *Tobii XR SDK Documentation*. Tobii.com.</ref>


* '''Raw data''' - Direct access to eye position coordinates and pupil measurements
### Data Access Levels
* '''Filtered data''' - Processed data with noise reduction and classification of eye movements
Developers typically access eye tracking data at different levels of abstraction:
* '''Semantic data''' - High-level interpretation of gaze targets and user attention<ref>Kumar, D., Dutta, A., Das, A., & Lahiri, U. (2016). SmartEye: Developing a novel eye tracking system for quantitative assessment of oculomotor abnormalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 24(10), 1051-1059.</ref>
*   '''Raw Data''': Direct coordinates of pupil centers, glints, eye openness, etc. Requires significant processing by the application.
*   '''Gaze Data''': Processed output providing calibrated gaze origin and direction vectors, or intersection points on virtual surfaces.
*  '''Eye Movement Events''': Classified data identifying fixations, saccades, and blinks.
*   '''Semantic Data''': Higher-level interpretations, such as the specific object being looked at (gaze target) or estimated attention levels.
<ref name="Kumar2016">Kumar, D., Dutta, A., Das, A., & Lahiri, U. (2016). SmartEye: Developing a novel eye tracking system for quantitative assessment of oculomotor abnormalities. *IEEE Transactions on Neural Systems and Rehabilitation Engineering, 24*(10), 1051-1059.</ref>


== Ethical and Privacy Considerations ==
== Ethical and Privacy Considerations ==


The powerful nature of eye tracking creates important ethical considerations:
The collection and use of eye tracking data necessitate careful ethical consideration:


* '''Biometric data protection''' - Eye tracking can create unique biometric signatures requiring appropriate safeguards
*   '''Biometric Data Privacy''': Gaze patterns can be unique identifiers. This data requires strong [[data security]] measures and compliance with regulations like [[GDPR]] (which may classify it as sensitive biometric data).
* '''Attention analytics''' - The potential for monitoring user attention raises privacy concerns
*   '''Inference and Profiling''': The potential to infer sensitive information (health, emotions, interests) without explicit consent raises ethical questions.
* '''Informed consent''' - Users should understand what eye data is collected and how it's used
*  '''Attention Monitoring''': Concerns about workplace surveillance or manipulative advertising based on attention analytics.
* '''Data minimization''' - Only necessary eye tracking data should be collected and stored<ref>Kröger, J. L., Lutz, O. H. M., & Müller, F. (2020). What does your gaze reveal about you? On the privacy implications of eye tracking. Privacy and Identity Management, 226-241.</ref>
*   '''Informed Consent''': Users must be clearly informed about what eye data is collected, how it is processed, stored, and shared, and for what purposes. Opt-in mechanisms are crucial.
*   '''Data Minimization and Anonymization''': Collect only necessary data, anonymize or aggregate whenever possible, and define clear data retention policies.
<ref name="Kroger2020"/>


== Standards and Regulations ==
== Standards and Regulations ==


Several organizations work on standardizing eye tracking technology:
Efforts are underway to standardize aspects of eye tracking technology and address its implications:
*  '''[[Khronos Group]] / [[OpenXR]]''': Defines standard APIs for accessing eye tracking data (e.g., `XR_EXT_eye_gaze_interaction`).
*  '''[[IEEE Standards Association|IEEE]]''': Working groups like IEEE P2048.5 focus on standards for immersive learning, potentially including eye tracking metrics.
*  '''[[Virtual Reality Industry Forum]] (VRIF)''': Develops guidelines for VR implementation, potentially covering eye tracking best practices.
*  '''Data Protection Regulations''': [[GDPR]] (Europe), [[CCPA]]/[[CPRA]] (California), and similar laws globally impose requirements on handling personal and biometric data, including eye tracking data.
*  '''[[ISO]]/[[IEC]] JTC 1/SC 24''': Committee working on international standards for computer graphics, image processing, and environmental data representation, relevant to VR/AR interfaces.<ref name="ISO_SC24">International Organization for Standardization. (2023). *ISO/IEC JTC 1/SC 24 - Computer graphics, image processing and environmental data representation*. ISO.org.</ref>


* '''[[IEEE P2048.5]]''' - Working group on eye tracking for VR/AR
== See Also ==
* '''[[VRIF Guidelines]]''' - Virtual Reality Industry Forum guidelines for eye tracking implementation
*  [[Augmented reality]]
* '''[[GDPR]]''' implications for eye tracking data in Europe
*  [[AR headset]]
* '''[[ISO/IEC JTC 1/SC 24]]''' - International standards for VR/AR interfaces including eye tracking<ref>International Organization for Standardization. (2021). ISO/IEC JTC 1/SC 24 - Computer graphics, image processing and environmental data representation. ISO.org.</ref>
*  [[Attention mapping]]
*  [[Avatar]]
*  [[Computer vision]]
*  [[Foveated rendering]]
*  [[Gaze]]
*   [[Hand tracking]]
*  [[Head-mounted display]]
*  [[Human eye]]
*  [[Human–computer interaction]]
*  [[Infrared]]
*  [[Interpupillary distance]]
*  [[Latency]]
*  [[Privacy]] in VR/AR
*   [[Pupil]]
*  [[Saccade]]
*  [[Social presence]]
*   [[Tobii]]
*   [[Varifocal display]]
*  [[Vergence-accommodation conflict]]
*  [[Virtual reality]]
*  [[VR headset]]


== References ==
== References ==
<references/>
<references />

Revision as of 16:09, 24 April 2025

  1. Eye Tracking

Eye tracking is a sensor technology that measures eye positions and eye movement. In the context of virtual reality (VR) and augmented reality (AR), it refers to the integration of sensors within VR headsets or AR headsets to determine precisely where the user is looking (gaze) in real-time within the virtual environment or overlaid digital interface. By accurately monitoring characteristics like pupil position, corneal reflections, and eye movements such as saccades and fixations, this technology enables more immersive, efficient, and intuitive user experiences. It has become an increasingly critical feature in modern head-mounted displays (HMDs), driving advancements in rendering, interaction, analytics, and social presence.[1]

History

While its integration into consumer VR/AR is relatively recent, the systematic study of eye movements dates back to the 19th century. Louis Émile Javal noted in 1879 that reading involved discrete stops (fixations) and rapid movements (saccades).[2] Early tracking devices included Edmund Huey's contact lens-based tracker (~1908)[3] and Guy Thomas Buswell's film-based systems in the 1930s. Alfred L. Yarbus's work in the 1960s highlighted how viewing patterns are task-dependent.[4] These foundational efforts paved the way for modern video-based and integrated tracking systems.

Technical Principles

Tracking Methods

Most eye tracking systems in contemporary VR and AR headsets employ one of several core technologies: 

   *   Illumination: Infrared (IR) light-emitting diodes (LEDs) safely illuminate the eye. IR light is used because it is invisible to the human eye, preventing distraction, and provides high contrast for cameras.
   *   Imaging: Small, high-frame rate infrared cameras capture images of the eye, specifically tracking the center of the pupil and the reflection(s) of the IR light off the surface of the cornea (known as glints).
   *   Algorithmic Processing: Sophisticated computer vision and image processing algorithms analyze the captured images. By calculating the vector between the pupil center and the corneal reflection(s), the system determines the eye's orientation and calculates the user's gaze point with high accuracy.
   *   Calibration: A per-user calibration process is usually required upon first use, and sometimes periodically, to account for individual differences in eye physiology (e.g., corneal shape, pupil size range) and the precise fit of the headset. This typically involves the user looking at specific points displayed within the headset.
  • Video-based eye tracking (Shape/Feature Tracking): Uses cameras aimed at the eyes to capture images, which are then analyzed using computer vision algorithms to identify eye features (pupil outline, iris texture, blood vessels) to determine gaze direction without necessarily relying on corneal reflections.[7] PCCR is often considered a subset of this broader category.
  • Electrooculography (EOG): Measures the electrical potential difference between electrodes placed on the skin around the eyes. This potential changes predictably as the eye rotates. While less common for high-accuracy gaze pointing in consumer VR/AR due to lower precision and susceptibility to muscle noise, it can be used for detecting larger eye movements or in specialized applications.[8]
  • Scleral search coil: Involves wearing a special contact lens containing a wire coil. The user sits within a magnetic field, and eye movements induce currents in the coil, providing extremely precise measurements. This is highly invasive and primarily used in laboratory research, not consumer VR/AR.[9]

Key Components (PCCR-based)

A typical PCCR eye tracking system integrated into a VR/AR headset consists of:

  • Illuminators: Infrared LEDs providing consistent, non-visible lighting.
  • Cameras: Specialized high-speed infrared cameras positioned to capture clear images of both eyes.
  • Processing Unit: Either onboard SoC resources or dedicated hardware to run the detection and gaze calculation algorithms in real-time.
  • Calibration Software: Software routines guiding the user through calibration and storing individual profiles.[10]

Eye Movement Types Measured

Eye tracking systems in VR/AR can detect and analyze various types of eye movements and states:

  • Fixations: Periods when the gaze remains relatively stable on a specific area (typically > 100-200 ms), indicating visual attention.
  • Saccades: Rapid, ballistic movements shifting the gaze between fixation points (typically 30-120 ms).
  • Smooth pursuit: Movements allowing the eyes to smoothly follow a moving object.
  • Vergence: The simultaneous movement of both eyes in opposite directions to obtain or maintain single binocular vision, critical for focusing on objects at different depths.
  • Pupil Size / Dilation: Changes in pupil diameter (pupillometry), which can correlate with changes in light levels, cognitive load, emotional arousal, or interest.[11]
  • Blinks: Detection of eyelid closures.

Applications in VR and AR

Eye tracking unlocks numerous capabilities enhancing VR and AR experiences:

  • Foveated rendering: Arguably the most impactful application for performance. By knowing precisely where the user is looking, the system renders the scene at maximum resolution only in the small, central area of the user's gaze (corresponding to the eye's high-acuity fovea), while rendering the peripheral areas at progressively lower resolutions. This mimics human vision and can drastically reduce the computational load on the graphics processing unit (GPU) – potentially by 30% to over 70% – allowing for higher fidelity graphics, increased frame rates, reduced latency, or lower power consumption, all without a noticeable loss in perceived visual quality.[12][13][14]
  • Natural Interaction / Gaze-Based Interaction: Eye tracking enables more intuitive control schemes:
   *   Gaze Selection/Pointing: Allows users to select objects, menu items, or targets simply by looking at them. This is often combined with a confirmation action like a button press on a controller, a hand tracking gesture (e.g., pinch), or a short dwell time.[15]
   *   Intent Prediction: Systems can anticipate user actions or needs based on gaze patterns (e.g., highlighting an object the user looks at intently).
   *   Gaze-directed Locomotion: Steering movement within the virtual world based on gaze direction.
  • Enhanced Social Presence / Avatar Realism: Realistic eye movements, including subtle saccades, blinks, and responsive gaze shifts, can be mirrored onto a user's avatar in social VR applications. This significantly enhances non-verbal communication and the feeling of social presence and connection when interacting with others.[16]
  • User Analytics and Research: Eye tracking provides invaluable objective data for:
   *   Usability testing & User Experience (UX) Research: Understanding how users visually explore and interact with interfaces or environments.
   *   Attention mapping: Creating heatmaps and gaze plots to visualize areas of interest and attention duration.
   *   Cognitive Load Assessment: Measuring mental workload through metrics like pupil dilation, blink rate, and fixation patterns.[17]
   *   Training and Simulation Analysis: Assessing trainee attention, situational awareness, and decision-making processes in professional simulations (e.g., medical, aviation).
  • Automatic IPD Adjustment: Some headsets utilize the eye tracking cameras to automatically measure the user's interpupillary distance (the distance between the centers of the pupils) and mechanically adjust the lens spacing for optimal visual clarity, stereo depth perception, and user comfort.
  • Accessibility: Eye tracking offers a powerful hands-free input modality for users with limited physical mobility, enabling them to navigate interfaces, communicate (e.g., gaze typing), and control applications within VR/AR.[18]
  • Adaptive Optics / Varifocal displays: Eye tracking is essential for dynamic varifocal displays, which adjust their focal plane based on where the user is looking in virtual depth. This helps address the vergence-accommodation conflict, potentially reducing eye strain and improving visual realism.[19]
  • Dynamic Distortion Compensation: Real-time adjustments to lens distortion correction based on precise eye position relative to the lens center can improve perceived sharpness across the field of view.[13]

Current Implementations

Several commercially available VR and AR headsets incorporate integrated eye tracking:

VR Headsets with Eye Tracking

  • Apple Vision Pro: Uses high-precision eye tracking as a primary input method (combined with hand gestures and voice), enabling UI navigation, selection, and foveated rendering.[20]
  • Meta Quest Pro: Features inward-facing sensors for eye and face tracking, primarily used for foveated rendering and driving realistic avatar expressions in social applications.[16]
  • PlayStation VR2: Integrates Tobii eye tracking technology for foveated rendering, gaze-based interactions in games, and enhanced immersion.[21]
  • HTC VIVE Pro Eye: An earlier enterprise-focused headset integrating Tobii eye tracking (accuracy ~0.5-1.1 degrees, 120Hz). Newer VIVE models may offer eye tracking via add-ons (e.g., VIVE Focus 3 Eye Tracker).[22]
  • Varjo XR-4, XR-3, VR-3, Aero: High-end professional headsets featuring industrial-grade eye tracking (sub-degree accuracy, 200Hz) for demanding simulation, research, and design applications.[23][24]
  • Pimax Crystal: Consumer-focused high-resolution headset incorporating eye tracking for features like foveated rendering and automatic IPD adjustment.[25]
  • Pico 4 Enterprise (previously Pico Neo 3 Pro Eye): Enterprise headsets integrating Tobii eye tracking for business applications.[26]
  • HP Reverb G2 Omnicept Edition: Featured integrated eye tracking (along with other biometric sensors) for enterprise use cases.[27]

AR Headsets with Eye Tracking

Eye tracking is also crucial for interaction and performance in AR:

  • Microsoft HoloLens 2: Uses eye tracking for user calibration, interaction (gaze targeting), and potentially performance optimization. Reported accuracy around 1.5 degrees.[28]
  • Magic Leap 2: Incorporates eye tracking for input (gaze, dwell), foveated rendering (on segmented dimmer), user calibration, and analytics.[29]

Technical Specifications and Performance Metrics

Key Performance Indicators

The quality and usability of eye tracking systems are measured using several critical metrics:

  • Accuracy: The average difference between the true gaze point and the system's reported gaze point, typically measured in degrees of visual angle. Good systems achieve 0.5° to 1.5° accuracy within the central field of view.
  • Precision: The consistency or reproducibility of measurements for the same gaze point (RMS of successive samples). Often between 0.1° and 0.5°.
  • Sampling Rate: The frequency at which eye position is measured, expressed in Hertz (Hz). Consumer systems range from 30Hz to 120Hz or more, while research systems can exceed 1000Hz. Higher rates capture more detail about rapid movements like saccades.
  • Latency: The time delay between an actual eye movement and when it is detected and reported by the system. Crucial for real-time applications like foveated rendering and interaction, ideally below 20ms, though system latency can sometimes be higher (e.g., 45-81 ms reported in some studies).[30]
  • Robustness / Tracking Ratio: The percentage of time the system successfully tracks the eyes under various conditions (e.g., different users, lighting, eyewear).
  • Field of View / Freedom of Head Movement: The range of eye rotation and head position within which the system maintains tracking.

[31]

Calibration Methods

Achieving specified accuracy typically requires individual user calibration:

  • Point Calibration: The most common method. The user looks sequentially at several points displayed on the screen while the system records corresponding eye data to build a mapping model.
  • Smooth Pursuit Calibration: The user follows one or more moving targets across the screen.
  • Implicit/Adjustment Calibration: Systems that attempt to calibrate or refine calibration based on natural viewing behavior during normal use, potentially reducing user friction.
  • Calibration Drift: Accuracy can degrade over time due to headset slippage or physiological changes, potentially requiring recalibration. Some studies note significant drift (e.g., 30% accuracy loss) within minutes under certain conditions.[30]

[32]

Challenges and Limitations

Despite significant progress, eye tracking in VR/AR still faces hurdles:

Technical Challenges

  • Individual Variations: Differences in eye physiology (e.g., corneal curvature, pupil size, eyelid shape, ethnicity-related features, "droopy" eyelids) can impact tracking accuracy and robustness.
  • Eyewear Compatibility: Prescription glasses (especially with thick lenses, bifocals, or certain coatings) and some types of contact lenses can interfere with IR illumination and camera imaging, degrading performance.
  • Processing Requirements: Real-time, high-frequency eye tracking requires significant computational resources, impacting overall system performance and battery life, especially on standalone mobile headsets.
  • Power Consumption: The cameras and illuminators continuously consume power, contributing to battery drain in untethered devices.
  • Accuracy/Latency Trade-offs: Achieving both high accuracy and very low latency simultaneously remains challenging.

[33]

User Experience Concerns

  • Calibration Process: The need for calibration can be perceived as inconvenient, especially if required frequently.
  • Privacy Implications: Eye tracking data is highly sensitive biometric data. It can potentially reveal information about identity, attention focus, cognitive state, emotional responses, interests, and even certain health conditions (e.g., fatigue, intoxication, neurological disorders), raising significant privacy concerns if collected, stored, or shared improperly.[34][35]
  • The "Uncanny Valley" Effect: Imperfectly synchronized or unnatural avatar eye movements can appear disturbing rather than enhance social presence.

Accessibility Issues

  • Compatibility with Eye Conditions: Certain ophthalmological conditions (e.g., strabismus, nystagmus, ptosis, corneal scarring) can significantly impair or prevent accurate eye tracking for affected individuals.
  • Make-up Interference: Certain types of mascara or eyeliner can sometimes interfere with tracking.

[36]

Future Developments

The field continues to evolve rapidly:

Emerging Technologies

  • Lower Power, Smaller Sensors: Miniaturization and improved power efficiency for better integration into lighter HMDs.
  • AI/ML-Enhanced Tracking: Using neural networks to improve robustness across diverse users, reduce calibration needs, and potentially infer more complex states from eye data.[37]
  • Advanced Optical Designs: Novel lens and camera configurations to improve tracking quality, especially for users with eyewear.
  • Sensor Fusion: Combining eye tracking data with other sensor inputs (e.g., electroencephalography (EEG), head tracking, facial expression tracking) for richer interaction and analysis.

Research Directions

  • Predictive Tracking: Algorithms that anticipate eye movements to compensate for system latency, enabling smoother foveated rendering and interaction.
  • Emotion and Cognitive State Recognition: Refining models to reliably infer affective states or cognitive load from eye metrics for adaptive interfaces or mental health applications.
  • Standardization: Developing common metrics, APIs, and data formats to facilitate cross-platform development and research comparison.
  • Longitudinal Tracking: Understanding how gaze patterns change over extended use and adapting systems accordingly.

[38]

Software Development and APIs

Integrating eye tracking into applications requires specific software support:

      1. Development Frameworks and APIs
  • Unity: Provides APIs through its XR Interaction Toolkit and potential vendor-specific SDKs (e.g., Tobii XR SDK).
  • Unreal Engine: Offers native interfaces and plugins for accessing eye tracking data.
  • OpenXR: The cross-platform standard includes an `XR_EXT_eye_gaze_interaction` extension, allowing developers to write more portable code.
  • Vendor SDKs: Companies like Tobii provide dedicated Software Development Kits offering fine-grained control and optimized features for their hardware.[39]
      1. Data Access Levels

Developers typically access eye tracking data at different levels of abstraction:

  • Raw Data: Direct coordinates of pupil centers, glints, eye openness, etc. Requires significant processing by the application.
  • Gaze Data: Processed output providing calibrated gaze origin and direction vectors, or intersection points on virtual surfaces.
  • Eye Movement Events: Classified data identifying fixations, saccades, and blinks.
  • Semantic Data: Higher-level interpretations, such as the specific object being looked at (gaze target) or estimated attention levels.

[40]

Ethical and Privacy Considerations

The collection and use of eye tracking data necessitate careful ethical consideration:

  • Biometric Data Privacy: Gaze patterns can be unique identifiers. This data requires strong data security measures and compliance with regulations like GDPR (which may classify it as sensitive biometric data).
  • Inference and Profiling: The potential to infer sensitive information (health, emotions, interests) without explicit consent raises ethical questions.
  • Attention Monitoring: Concerns about workplace surveillance or manipulative advertising based on attention analytics.
  • Informed Consent: Users must be clearly informed about what eye data is collected, how it is processed, stored, and shared, and for what purposes. Opt-in mechanisms are crucial.
  • Data Minimization and Anonymization: Collect only necessary data, anonymize or aggregate whenever possible, and define clear data retention policies.

[34]

Standards and Regulations

Efforts are underway to standardize aspects of eye tracking technology and address its implications:

  • Khronos Group / OpenXR: Defines standard APIs for accessing eye tracking data (e.g., `XR_EXT_eye_gaze_interaction`).
  • IEEE: Working groups like IEEE P2048.5 focus on standards for immersive learning, potentially including eye tracking metrics.
  • Virtual Reality Industry Forum (VRIF): Develops guidelines for VR implementation, potentially covering eye tracking best practices.
  • Data Protection Regulations: GDPR (Europe), CCPA/CPRA (California), and similar laws globally impose requirements on handling personal and biometric data, including eye tracking data.
  • ISO/IEC JTC 1/SC 24: Committee working on international standards for computer graphics, image processing, and environmental data representation, relevant to VR/AR interfaces.[41]

See Also

References

  1. ↑ Duchowski, A. T. (2017). *Eye Tracking Methodology: Theory and Practice*. Springer.
  2. ↑ Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. *Psychological Bulletin, 124*(3), 372–422.
  3. ↑ Huey, E.B. (1908). *The Psychology and Pedagogy of Reading*. Macmillan.
  4. ↑ Yarbus, A.L. (1967). *Eye Movements and Vision*. Plenum Press.
  5. ↑ Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). *Eye tracking: A comprehensive guide to methods and measures*. Oxford University Press.
  6. ↑ Lang, B. (2021, July 21). *Casual Explainer: What is Eye-tracking & How Does it Work?* Road to VR. https://www.roadtovr.com/casual-explainer-what-is-eye-tracking-how-does-it-work/
  7. ↑ Hansen, D. W., & Ji, Q. (2010). In the eye of the beholder: A survey of models for eyes and gaze. *IEEE Transactions on Pattern Analysis and Machine Intelligence, 32*(3), 478-500.
  8. ↑ Bulling, A., Ward, J. A., Gellersen, H., & Tröster, G. (2011). Eye movement analysis for activity recognition using electrooculography. *IEEE Transactions on Pattern Analysis and Machine Intelligence, 33*(4), 741-753.
  9. ↑ Robinson, D.A. (1963). A method of measuring eye movement using a scleral search coil in a magnetic field. *IEEE Transactions on Biomedical Engineering, BME-10*(4), 137–145.
  10. ↑ Kar, A., & Corcoran, P. (2017). A review and analysis of eye-gaze estimation systems, algorithms and performance evaluation methods in consumer platforms. *IEEE Access, 5*, 16495-16519.
  11. ↑ Leigh, R. J., & Zee, D. S. (2015). *The neurology of eye movements*. Oxford University Press.
  12. ↑ Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C., Benty, N., Luebke, D., & Lefohn, A. (2016). Towards foveated rendering for gaze-tracked virtual reality. *ACM Transactions on Graphics, 35*(6), 179.
  13. ↑ 13.0 13.1 Tobii Blog: Eye Tracking in VR — A Vital Component. Retrieved Nov 17, 2023.
  14. ↑ NVIDIA Corporation. (n.d.). *Maximize VR Performance with Foveated Rendering*. NVIDIA Developer. Retrieved November 16, 2023, from https://developer.nvidia.com/vrworks/graphics/foveatedrendering
  15. ↑ Piumsomboon, T., Lee, G., Lindeman, R. W., & Billinghurst, M. (2017). Exploring natural eye-gaze-based interaction for immersive virtual reality. *IEEE Symposium on 3D User Interfaces*, 36-39.
  16. ↑ 16.0 16.1 Meta Platforms, Inc. (2022, October 11). *Meta Quest Pro: A New Way to Work, Create and Collaborate*. Meta Quest Blog. https://www.meta.com/blog/quest/meta-quest-pro-vr-headset-features-price-release-date/
  17. ↑ Clay, V., König, P., & König, S. (2019). Eye tracking in virtual reality. *Journal of Eye Movement Research, 12*(1).
  18. ↑ Yuan, Z., Bi, T., Muntean, G. M., & Ghinea, G. (2020). Perceived synchronization of mulsemedia services. *IEEE Transactions on Multimedia, 17*(7), 957-966. Note: While this ref discusses synchronization, accessibility is a widely cited benefit of eye tracking.
  19. ↑ Akeley, K., Watt, S.J., Girshick, A.R., & Banks, M.S. (2004). A stereo display prototype with multiple focal distances. *ACM Transactions on Graphics, 23*(3), 804–813.
  20. ↑ Apple Inc. (n.d.). *Apple Vision Pro*. Apple. Retrieved November 16, 2023, from https://www.apple.com/apple-vision-pro/
  21. ↑ Sony Interactive Entertainment. (2023). *PS VR2 Features*. PlayStation.com. https://www.playstation.com/en-us/ps-vr2/features/
  22. ↑ HTC Corporation. (n.d.). *VIVE Pro Eye*. VIVE. Retrieved November 16, 2023, from https://www.vive.com/us/product/vive-pro-eye/overview/
  23. ↑ Varjo Technologies Oy. (n.d.). *Varjo Aero*. Varjo. Retrieved November 16, 2023, from https://varjo.com/products/aero/
  24. ↑ Varjo Technologies. (2021). *Varjo XR-3 Technical Specifications*. Varjo.com.
  25. ↑ Pimax Technology (Shanghai) Co., Ltd. (n.d.). *Pimax Crystal*. Pimax. Retrieved November 16, 2023, from https://pimax.com/crystal/
  26. ↑ Pico Interactive. (2021). *Pico Neo 3 Pro Eye Specifications*. Pico-interactive.com.
  27. ↑ HP Development Company, L.P. (n.d.). *HP Reverb G2 Omnicept Edition VR Headset*. HP.com. Retrieved November 16, 2023, from https://www.hp.com/us-en/vr/reverb-g2-vr-headset-omnicept-edition.html
  28. ↑ Microsoft Corporation. (2019). *HoloLens 2 Hardware Details*. Microsoft.com.
  29. ↑ Magic Leap, Inc. (2022). *Magic Leap 2 Technical Overview*. MagicLeap.com.
  30. ↑ 30.0 30.1 Mack, S., et al. (2022). A survey on eye tracking in virtual and augmented reality. *Virtual Reality*, 27, 1597–1625. https://link.springer.com/article/10.1007/s10055-022-00738-z
  31. ↑ Blignaut, P. (2018). Using eye tracking to assess user experience: A case of a mobile banking application. In *ACM International Conference Proceeding Series*, 219-228.
  32. ↑ Santini, T., Fuhl, W., & Kasneci, E. (2018). CalibMe: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction. *CHI Conference on Human Factors in Computing Systems*, 1-6.
  33. ↑ Majaranta, P., & Bulling, A. (2014). Eye tracking and eye-based human-computer interaction. In *Advances in physiological computing*, 39-65. Springer.
  34. ↑ 34.0 34.1 Kröger, J. L., Lutz, O. H. M., & Müller, F. (2020). What does your gaze reveal about you? On the privacy implications of eye tracking. *Privacy and Identity Management*, 226-241.
  35. ↑ Crockford, K., & Electronic Frontier Foundation. (2020, November 19). *The Privacy Bird Isn't Real: Your VR/AR Data Is*. Electronic Frontier Foundation. https://www.eff.org/deeplinks/2020/11/privacy-bird-isnt-real-your-vrar-data
  36. ↑ Titz, J., Scholz, A., & Sedlmeier, P. (2018). Comparing eye trackers by correlating their eye-metric data. *Behavior Research Methods, 50*(5), 1853-1863.
  37. ↑ Fuhl, W., Santini, T., Kasneci, G., & Kasneci, E. (2016). PupilNet: Convolutional neural networks for robust pupil detection. *CoRR, abs/1601.04902*.
  38. ↑ Duchowski, A. T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M., & Giannopoulos, I. (2018). The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation. *CHI Conference on Human Factors in Computing Systems*, 1-13.
  39. ↑ Tobii Technology. (2023). *Tobii XR SDK Documentation*. Tobii.com.
  40. ↑ Kumar, D., Dutta, A., Das, A., & Lahiri, U. (2016). SmartEye: Developing a novel eye tracking system for quantitative assessment of oculomotor abnormalities. *IEEE Transactions on Neural Systems and Rehabilitation Engineering, 24*(10), 1051-1059.
  41. ↑ International Organization for Standardization. (2023). *ISO/IEC JTC 1/SC 24 - Computer graphics, image processing and environmental data representation*. ISO.org.
Retrieved from "https://vrarwiki.com/index.php?title=Eye_tracking&oldid=34455"