Eye tracking: Difference between revisions
Xinreality (talk | contribs) |
Xinreality (talk | contribs) No edit summary |
||
| Line 1: | Line 1: | ||
{{see also|Terms|Technical Terms}} | |||
[[Eye tracking]] is a sensor technology that measures [[eye]] positions and [[eye movement]]. In the context of [[virtual reality]] (VR) and [[augmented reality]] (AR), it refers to the integration of sensors within [[VR headset]]s or [[AR headset]]s to determine precisely where the user is looking ([[gaze]]) in real-time within the [[virtual environment]] or overlaid digital interface. By accurately monitoring characteristics like pupil position, corneal reflections, and eye movements such as [[saccades]] and fixations, this technology enables more immersive, efficient, and intuitive user experiences. It has become an increasingly critical feature in modern [[head-mounted display]]s ([[HMD]]s), driving advancements in rendering, interaction, analytics, and social presence.<ref name="Duchowski2017">Duchowski, A. T. (2017). *Eye Tracking Methodology: Theory and Practice*. Springer.</ref> | |||
== History == | ==History== | ||
While its integration into consumer VR/AR is relatively recent, the systematic study of eye movements dates back to the 19th century. [[Louis Émile Javal]] noted in 1879 that reading involved discrete stops (fixations) and rapid movements (saccades).<ref name="Rayner1998">Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. *Psychological Bulletin, 124*(3), 372–422.</ref> Early tracking devices included [[Edmund Huey]]'s contact lens-based tracker (~1908)<ref name="Huey1908">Huey, E.B. (1908). *The Psychology and Pedagogy of Reading*. Macmillan.</ref> and [[Guy Thomas Buswell]]'s film-based systems in the 1930s. [[Alfred L. Yarbus]]'s work in the 1960s highlighted how viewing patterns are task-dependent.<ref name="Yarbus1967">Yarbus, A.L. (1967). *Eye Movements and Vision*. Plenum Press.</ref> These foundational efforts paved the way for modern video-based and integrated tracking systems. | While its integration into consumer VR/AR is relatively recent, the systematic study of eye movements dates back to the 19th century. [[Louis Émile Javal]] noted in 1879 that reading involved discrete stops (fixations) and rapid movements (saccades).<ref name="Rayner1998">Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. *Psychological Bulletin, 124*(3), 372–422.</ref> Early tracking devices included [[Edmund Huey]]'s contact lens-based tracker (~1908)<ref name="Huey1908">Huey, E.B. (1908). *The Psychology and Pedagogy of Reading*. Macmillan.</ref> and [[Guy Thomas Buswell]]'s film-based systems in the 1930s. [[Alfred L. Yarbus]]'s work in the 1960s highlighted how viewing patterns are task-dependent.<ref name="Yarbus1967">Yarbus, A.L. (1967). *Eye Movements and Vision*. Plenum Press.</ref> These foundational efforts paved the way for modern video-based and integrated tracking systems. | ||
==Technical Principles== | ==Technical Principles== | ||
===Tracking Method=== | ===Tracking Method=== | ||
Most eye tracking systems in contemporary VR and AR headsets employ one of several core technologies: | Most eye tracking systems in contemporary VR and AR headsets employ one of several core technologies: | ||
| Line 63: | Line 63: | ||
*'''[[Dynamic Distortion Compensation]]''': Real-time adjustments to lens distortion correction based on precise eye position relative to the lens center can improve perceived sharpness across the field of view.<ref name="TobiiFoveated"/> | *'''[[Dynamic Distortion Compensation]]''': Real-time adjustments to lens distortion correction based on precise eye position relative to the lens center can improve perceived sharpness across the field of view.<ref name="TobiiFoveated"/> | ||
== Current Implementations == | ==Current Implementations== | ||
Several commercially available VR and AR headsets incorporate integrated eye tracking: | Several commercially available VR and AR headsets incorporate integrated eye tracking: | ||
=== VR Headsets with Eye Tracking === | ===VR Headsets with Eye Tracking=== | ||
* | *'''[[Apple Vision Pro]]''': Uses high-precision eye tracking as a primary input method (combined with hand gestures and voice), enabling UI navigation, selection, and foveated rendering.<ref name="AppleVisionPro">Apple Inc. (n.d.). *Apple Vision Pro*. Apple. Retrieved November 16, 2023, from https://www.apple.com/apple-vision-pro/</ref> | ||
* | *'''[[Meta Quest Pro]]''': Features inward-facing sensors for eye and face tracking, primarily used for foveated rendering and driving realistic avatar expressions in social applications.<ref name="MetaAvatarsEyeTracking"/> | ||
* | *'''[[PlayStation VR2]]''': Integrates [[Tobii]] eye tracking technology for foveated rendering, gaze-based interactions in games, and enhanced immersion.<ref name="SonyPSVR2">Sony Interactive Entertainment. (2023). *PS VR2 Features*. PlayStation.com. https://www.playstation.com/en-us/ps-vr2/features/</ref> | ||
* | *'''[[HTC VIVE Pro Eye]]''': An earlier enterprise-focused headset integrating Tobii eye tracking (accuracy ~0.5-1.1 degrees, 120Hz). Newer VIVE models may offer eye tracking via add-ons (e.g., VIVE Focus 3 Eye Tracker).<ref name="ViveProEye">HTC Corporation. (n.d.). *VIVE Pro Eye*. VIVE. Retrieved November 16, 2023, from https://www.vive.com/us/product/vive-pro-eye/overview/</ref> | ||
* | *'''[[Varjo]] XR-4, XR-3, VR-3, Aero''': High-end professional headsets featuring industrial-grade eye tracking (sub-degree accuracy, 200Hz) for demanding simulation, research, and design applications.<ref name="VarjoAero">Varjo Technologies Oy. (n.d.). *Varjo Aero*. Varjo. Retrieved November 16, 2023, from https://varjo.com/products/aero/</ref><ref name="VarjoXR3">Varjo Technologies. (2021). *Varjo XR-3 Technical Specifications*. Varjo.com.</ref> | ||
* | *'''[[Pimax]] Crystal''': Consumer-focused high-resolution headset incorporating eye tracking for features like foveated rendering and automatic IPD adjustment.<ref name="PimaxCrystal">Pimax Technology (Shanghai) Co., Ltd. (n.d.). *Pimax Crystal*. Pimax. Retrieved November 16, 2023, from https://pimax.com/crystal/</ref> | ||
* | *'''[[Pico 4 Enterprise]]''' (previously Pico Neo 3 Pro Eye): Enterprise headsets integrating Tobii eye tracking for business applications.<ref name="PicoNeo3Eye">Pico Interactive. (2021). *Pico Neo 3 Pro Eye Specifications*. Pico-interactive.com.</ref> | ||
* | *'''[[HP Reverb G2 Omnicept Edition]]''': Featured integrated eye tracking (along with other biometric sensors) for enterprise use cases.<ref name="HPOmnicept">HP Development Company, L.P. (n.d.). *HP Reverb G2 Omnicept Edition VR Headset*. HP.com. Retrieved November 16, 2023, from https://www.hp.com/us-en/vr/reverb-g2-vr-headset-omnicept-edition.html</ref> | ||
=== AR Headsets with Eye Tracking === | ===AR Headsets with Eye Tracking=== | ||
Eye tracking is also crucial for interaction and performance in AR: | Eye tracking is also crucial for interaction and performance in AR: | ||
* | *'''[[Microsoft HoloLens 2]]''': Uses eye tracking for user calibration, interaction (gaze targeting), and potentially performance optimization. Reported accuracy around 1.5 degrees.<ref name="HoloLens2">Microsoft Corporation. (2019). *HoloLens 2 Hardware Details*. Microsoft.com.</ref> | ||
* | *'''[[Magic Leap 2]]''': Incorporates eye tracking for input (gaze, dwell), foveated rendering (on segmented dimmer), user calibration, and analytics.<ref name="MagicLeap2">Magic Leap, Inc. (2022). *Magic Leap 2 Technical Overview*. MagicLeap.com.</ref> | ||
== Technical Specifications and Performance Metrics == | ==Technical Specifications and Performance Metrics== | ||
===Key Performance Indicators=== | |||
=== Key Performance Indicators === | |||
The quality and usability of eye tracking systems are measured using several critical metrics: | The quality and usability of eye tracking systems are measured using several critical metrics: | ||
* | *'''Accuracy''': The average difference between the true gaze point and the system's reported gaze point, typically measured in degrees of visual angle. Good systems achieve 0.5° to 1.5° accuracy within the central field of view. | ||
* | *'''Precision''': The consistency or reproducibility of measurements for the same gaze point (RMS of successive samples). Often between 0.1° and 0.5°. | ||
* | *'''Sampling Rate''': The frequency at which eye position is measured, expressed in Hertz (Hz). Consumer systems range from 30Hz to 120Hz or more, while research systems can exceed 1000Hz. Higher rates capture more detail about rapid movements like saccades. | ||
* | *'''Latency''': The time delay between an actual eye movement and when it is detected and reported by the system. Crucial for real-time applications like foveated rendering and interaction, ideally below 20ms, though system latency can sometimes be higher (e.g., 45-81 ms reported in some studies).<ref name="SpringerReview2022">Mack, S., et al. (2022). A survey on eye tracking in virtual and augmented reality. *Virtual Reality*, 27, 1597–1625. https://link.springer.com/article/10.1007/s10055-022-00738-z</ref> | ||
* | *'''Robustness / Tracking Ratio''': The percentage of time the system successfully tracks the eyes under various conditions (e.g., different users, lighting, eyewear). | ||
* | *'''Field of View / Freedom of Head Movement''': The range of eye rotation and head position within which the system maintains tracking. | ||
<ref name="Blignaut2018">Blignaut, P. (2018). Using eye tracking to assess user experience: A case of a mobile banking application. In *ACM International Conference Proceeding Series*, 219-228.</ref> | <ref name="Blignaut2018">Blignaut, P. (2018). Using eye tracking to assess user experience: A case of a mobile banking application. In *ACM International Conference Proceeding Series*, 219-228.</ref> | ||
=== Calibration Methods === | ===Calibration Methods=== | ||
Achieving specified accuracy typically requires individual user calibration: | Achieving specified accuracy typically requires individual user calibration: | ||
* | *'''Point Calibration''': The most common method. The user looks sequentially at several points displayed on the screen while the system records corresponding eye data to build a mapping model. | ||
* | *'''Smooth Pursuit Calibration''': The user follows one or more moving targets across the screen. | ||
* | *'''Implicit/Adjustment Calibration''': Systems that attempt to calibrate or refine calibration based on natural viewing behavior during normal use, potentially reducing user friction. | ||
* | *'''Calibration Drift''': Accuracy can degrade over time due to headset slippage or physiological changes, potentially requiring recalibration. Some studies note significant drift (e.g., 30% accuracy loss) within minutes under certain conditions.<ref name="SpringerReview2022"/> | ||
<ref name="Santini2018">Santini, T., Fuhl, W., & Kasneci, E. (2018). CalibMe: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction. *CHI Conference on Human Factors in Computing Systems*, 1-6.</ref> | <ref name="Santini2018">Santini, T., Fuhl, W., & Kasneci, E. (2018). CalibMe: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction. *CHI Conference on Human Factors in Computing Systems*, 1-6.</ref> | ||
== Challenges and Limitations == | ==Challenges and Limitations== | ||
Despite significant progress, eye tracking in VR/AR still faces hurdles: | Despite significant progress, eye tracking in VR/AR still faces hurdles: | ||
=== Technical Challenges === | ===Technical Challenges=== | ||
* | *'''Individual Variations''': Differences in eye physiology (e.g., corneal curvature, pupil size, eyelid shape, ethnicity-related features, "droopy" eyelids) can impact tracking accuracy and robustness. | ||
* | *'''Eyewear Compatibility''': Prescription glasses (especially with thick lenses, bifocals, or certain coatings) and some types of contact lenses can interfere with IR illumination and camera imaging, degrading performance. | ||
* | *'''Processing Requirements''': Real-time, high-frequency eye tracking requires significant computational resources, impacting overall system performance and battery life, especially on standalone mobile headsets. | ||
* | *'''Power Consumption''': The cameras and illuminators continuously consume power, contributing to battery drain in untethered devices. | ||
* | *'''Accuracy/Latency Trade-offs''': Achieving both high accuracy and very low latency simultaneously remains challenging. | ||
<ref name="Majaranta2014">Majaranta, P., & Bulling, A. (2014). Eye tracking and eye-based human-computer interaction. In *Advances in physiological computing*, 39-65. Springer.</ref> | <ref name="Majaranta2014">Majaranta, P., & Bulling, A. (2014). Eye tracking and eye-based human-computer interaction. In *Advances in physiological computing*, 39-65. Springer.</ref> | ||
=== User Experience Concerns === | ===User Experience Concerns=== | ||
* | *'''Calibration Process''': The need for calibration can be perceived as inconvenient, especially if required frequently. | ||
* | *'''[[Privacy]] Implications''': Eye tracking data is highly sensitive [[biometric data]]. It can potentially reveal information about identity, attention focus, cognitive state, emotional responses, interests, and even certain health conditions (e.g., fatigue, intoxication, neurological disorders), raising significant privacy concerns if collected, stored, or shared improperly.<ref name="Kroger2020">Kröger, J. L., Lutz, O. H. M., & Müller, F. (2020). What does your gaze reveal about you? On the privacy implications of eye tracking. *Privacy and Identity Management*, 226-241.</ref><ref name="EFFPrivacyVRAR">Crockford, K., & Electronic Frontier Foundation. (2020, November 19). *The Privacy Bird Isn't Real: Your VR/AR Data Is*. Electronic Frontier Foundation. https://www.eff.org/deeplinks/2020/11/privacy-bird-isnt-real-your-vrar-data</ref> | ||
* | *'''The "[[Uncanny Valley]]" Effect''': Imperfectly synchronized or unnatural avatar eye movements can appear disturbing rather than enhance social presence. | ||
=== Accessibility Issues === | ===Accessibility Issues=== | ||
* | *'''Compatibility with Eye Conditions''': Certain ophthalmological conditions (e.g., strabismus, nystagmus, ptosis, corneal scarring) can significantly impair or prevent accurate eye tracking for affected individuals. | ||
* | *'''Make-up Interference''': Certain types of mascara or eyeliner can sometimes interfere with tracking. | ||
<ref name="Titz2018">Titz, J., Scholz, A., & Sedlmeier, P. (2018). Comparing eye trackers by correlating their eye-metric data. *Behavior Research Methods, 50*(5), 1853-1863.</ref> | <ref name="Titz2018">Titz, J., Scholz, A., & Sedlmeier, P. (2018). Comparing eye trackers by correlating their eye-metric data. *Behavior Research Methods, 50*(5), 1853-1863.</ref> | ||
== Future Developments == | ==Future Developments== | ||
The field continues to evolve rapidly: | The field continues to evolve rapidly: | ||
=== Emerging Technologies === | ===Emerging Technologies=== | ||
* | *'''Lower Power, Smaller Sensors''': Miniaturization and improved power efficiency for better integration into lighter HMDs. | ||
* | *'''[[Artificial intelligence|AI]]/[[Machine learning|ML]]-Enhanced Tracking''': Using neural networks to improve robustness across diverse users, reduce calibration needs, and potentially infer more complex states from eye data.<ref name="Fuhl2016">Fuhl, W., Santini, T., Kasneci, G., & Kasneci, E. (2016). PupilNet: Convolutional neural networks for robust pupil detection. *CoRR, abs/1601.04902*.</ref> | ||
* | *'''Advanced Optical Designs''': Novel lens and camera configurations to improve tracking quality, especially for users with eyewear. | ||
* | *'''Sensor Fusion''': Combining eye tracking data with other sensor inputs (e.g., [[electroencephalography]] (EEG), head tracking, facial expression tracking) for richer interaction and analysis. | ||
=== Research Directions === | ===Research Directions=== | ||
* | *'''Predictive Tracking''': Algorithms that anticipate eye movements to compensate for system latency, enabling smoother foveated rendering and interaction. | ||
* | *'''Emotion and Cognitive State Recognition''': Refining models to reliably infer affective states or cognitive load from eye metrics for adaptive interfaces or mental health applications. | ||
* | *'''Standardization''': Developing common metrics, APIs, and data formats to facilitate cross-platform development and research comparison. | ||
* | *'''Longitudinal Tracking''': Understanding how gaze patterns change over extended use and adapting systems accordingly. | ||
<ref name="Duchowski2018">Duchowski, A. T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M., & Giannopoulos, I. (2018). The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation. *CHI Conference on Human Factors in Computing Systems*, 1-13.</ref> | <ref name="Duchowski2018">Duchowski, A. T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M., & Giannopoulos, I. (2018). The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation. *CHI Conference on Human Factors in Computing Systems*, 1-13.</ref> | ||