Hand tracking: Difference between revisions
Xinreality (talk | contribs) No edit summary |
Xinreality (talk | contribs) m Text replacement - "e.g.," to "for example" Tags: Mobile edit Mobile web edit |
||
| Line 15: | Line 15: | ||
'''Hand tracking''' is a [[computer vision]]-based technology used in [[virtual reality]] (VR), [[augmented reality]] (AR), and [[mixed reality]] (MR) systems to detect, track, and interpret the position, orientation, and movements of a user's hands and fingers in real time. Unlike traditional input methods such as [[motion controller|controllers]] or gloves, hand tracking enables controller-free, natural interactions by leveraging cameras, sensors, and artificial intelligence (AI) algorithms to map hand poses into virtual environments.<ref name="Frontiers2021" /> This technology enhances immersion, presence, and usability in [[extended reality]] (XR) applications by allowing users to perform gestures like pointing, grabbing, pinching, and swiping directly with their bare hands. | '''Hand tracking''' is a [[computer vision]]-based technology used in [[virtual reality]] (VR), [[augmented reality]] (AR), and [[mixed reality]] (MR) systems to detect, track, and interpret the position, orientation, and movements of a user's hands and fingers in real time. Unlike traditional input methods such as [[motion controller|controllers]] or gloves, hand tracking enables controller-free, natural interactions by leveraging cameras, sensors, and artificial intelligence (AI) algorithms to map hand poses into virtual environments.<ref name="Frontiers2021" /> This technology enhances immersion, presence, and usability in [[extended reality]] (XR) applications by allowing users to perform gestures like pointing, grabbing, pinching, and swiping directly with their bare hands. | ||
Hand tracking systems typically operate using optical methods, such as [[infrared]] (IR) illumination and monochrome cameras, or visible-light cameras integrated into [[head-mounted display]]s (HMDs). Modern implementations achieve low-latency tracking ( | Hand tracking systems typically operate using optical methods, such as [[infrared]] (IR) illumination and monochrome cameras, or visible-light cameras integrated into [[head-mounted display]]s (HMDs). Modern implementations achieve low-latency tracking (for example 10–70 ms) with high accuracy, supporting up to 27 degrees of freedom (DoF) per hand to capture complex articulations.<ref name="UltraleapDocs" /> The human hand has approximately 27 degrees of freedom, making accurate tracking a complex challenge.<ref name="HandDoF" /> It has evolved from early wired prototypes in the 1970s to sophisticated, software-driven solutions integrated into consumer devices like the [[Meta Quest]] series, [[Microsoft HoloLens 2]], and [[Apple Vision Pro]]. | ||
Hand tracking is a cornerstone of [[human-computer interaction]] in [[spatial computing]]. Modern systems commonly provide a per-hand skeletal pose ( | Hand tracking is a cornerstone of [[human-computer interaction]] in [[spatial computing]]. Modern systems commonly provide a per-hand skeletal pose (for example joints and bones), expose this data to applications through standard APIs (such as [[OpenXR]] and [[WebXR]]), and pair it with higher-level interaction components (for example poke, grab, raycast) for robust user experiences across devices.<ref name="OpenXR11" /><ref name="WebXRHand" /> | ||
== History == | == History == | ||
| Line 50: | Line 50: | ||
=== 2000s: Sensor Fusion and Early Commercialization === | === 2000s: Sensor Fusion and Early Commercialization === | ||
The 2000s saw the convergence of hardware and software for multi-modal tracking. External devices like data gloves with fiber-optic sensors ( | The 2000s saw the convergence of hardware and software for multi-modal tracking. External devices like data gloves with fiber-optic sensors (for example Fifth Dimension Technologies' 5DT Glove) combined bend sensors with IMUs to capture 3D hand poses. Software frameworks began processing fused data for virtual hand avatars. However, these remained bulky and controller-dependent, with limited adoption outside research labs.<ref name="VirtualSpeech" /> | ||
In the late 1990s and early 2000s, camera-based gesture recognition began to be explored outside of VR, for instance, computer vision researchers worked on interpreting hand signs for sign language or basic gesture control of computers. However, real-time markerless hand tracking in 3D was extremely challenging with the processing power then available. | In the late 1990s and early 2000s, camera-based gesture recognition began to be explored outside of VR, for instance, computer vision researchers worked on interpreting hand signs for sign language or basic gesture control of computers. However, real-time markerless hand tracking in 3D was extremely challenging with the processing power then available. | ||
| Line 129: | Line 129: | ||
Some systems augment or replace optical tracking with active depth sensing such as [[LiDAR]] or structured light infrared systems. These emit light (laser or IR LED) and measure its reflection to more precisely determine the distance and shape of hands, even in low-light conditions. LiDAR-based hand tracking can capture 3D positions with high precision and is less affected by ambient lighting or distance than pure camera-based methods.<ref name="VRExpert2023" /> | Some systems augment or replace optical tracking with active depth sensing such as [[LiDAR]] or structured light infrared systems. These emit light (laser or IR LED) and measure its reflection to more precisely determine the distance and shape of hands, even in low-light conditions. LiDAR-based hand tracking can capture 3D positions with high precision and is less affected by ambient lighting or distance than pure camera-based methods.<ref name="VRExpert2023" /> | ||
Ultraleap's hand tracking module ( | Ultraleap's hand tracking module (for example the Stereo IR 170 sensor) projects IR light and uses two IR cameras to track hands in 3D, allowing for robust tracking under various lighting conditions. This module has been integrated into devices like the Varjo VR-3/XR-3 and certain [[Pico]] headsets to provide built-in hand tracking.<ref name="SoundxVision" /><ref name="VRExpert2023" /> Active depth systems (for example [[time-of-flight camera|Time-of-Flight]] or [[structured light]]) project or emit IR to recover per-pixel depth, improving robustness in low light and during complex hand poses. Several headsets integrate IR illumination to make hands stand out for monochrome sensors. Some [[mixed reality]] devices also include dedicated scene depth sensors that aid perception and interaction. | ||
Optical hand tracking is generally affordable to implement since it can leverage the same camera hardware used for environment tracking or passthrough video. However, its performance can be affected by the cameras' field of view, lighting conditions, and frame rate. If the user's hands move outside the view of the cameras or lighting is poor, tracking quality will suffer. Improvements in computer vision and AI have steadily increased the accuracy and robustness of optical hand tracking, enabling features like two-hand interactions and fine finger gesture detection.<ref name="VRExpert2023" /> | Optical hand tracking is generally affordable to implement since it can leverage the same camera hardware used for environment tracking or passthrough video. However, its performance can be affected by the cameras' field of view, lighting conditions, and frame rate. If the user's hands move outside the view of the cameras or lighting is poor, tracking quality will suffer. Improvements in computer vision and AI have steadily increased the accuracy and robustness of optical hand tracking, enabling features like two-hand interactions and fine finger gesture detection.<ref name="VRExpert2023" /> | ||
| Line 141: | Line 141: | ||
* '''[[OpenXR]]''': A cross-vendor API from the Khronos Group. Version 1.1 (April 2024) consolidated hand tracking into the core specification, folding common extensions and providing standardized hand-tracking data structures and joint hierarchies across devices, easing portability for developers. The XR_EXT_hand_tracking extension provides 26 joint locations with standardized hierarchy.<ref name="OpenXR11" /> | * '''[[OpenXR]]''': A cross-vendor API from the Khronos Group. Version 1.1 (April 2024) consolidated hand tracking into the core specification, folding common extensions and providing standardized hand-tracking data structures and joint hierarchies across devices, easing portability for developers. The XR_EXT_hand_tracking extension provides 26 joint locations with standardized hierarchy.<ref name="OpenXR11" /> | ||
* '''[[WebXR]] Hand Input Module''' (W3C): The Level 1 specification represents the W3C standard for browser-based hand tracking, enabling web applications to access articulated hand pose data ( | * '''[[WebXR]] Hand Input Module''' (W3C): The Level 1 specification represents the W3C standard for browser-based hand tracking, enabling web applications to access articulated hand pose data (for example joint poses) so web apps can implement hands-first interaction.<ref name="WebXRHand" /> | ||
== Notable Platforms == | == Notable Platforms == | ||
| Line 153: | Line 153: | ||
| [[Apple Vision Pro]] || Multi-camera, IR illumination, [[LiDAR]] scene sensing; eye-hand fusion || "Look to target, pinch to select", flick to scroll; relaxed, low-effort micro-gestures || Hand + eye as primary input paradigm in visionOS<ref name="AppleGestures" /> | | [[Apple Vision Pro]] || Multi-camera, IR illumination, [[LiDAR]] scene sensing; eye-hand fusion || "Look to target, pinch to select", flick to scroll; relaxed, low-effort micro-gestures || Hand + eye as primary input paradigm in visionOS<ref name="AppleGestures" /> | ||
|- | |- | ||
| [[Ultraleap]] modules ( | | [[Ultraleap]] modules (for example Controller 2, Stereo IR) || Stereo IR + LEDs; skeletal model || Robust two-hand support; integrations for Unity/Unreal/OpenXR || Widely embedded in enterprise headsets (for example Varjo XR-3/VR-3)<ref name="UltraleapDocs" /><ref name="VarjoUltraleap" /> | ||
|} | |} | ||
| Line 169: | Line 169: | ||
=== [[Ray-Based Selection]] (Indirect Interaction) === | === [[Ray-Based Selection]] (Indirect Interaction) === | ||
For distant objects beyond arm's reach, a virtual ray (from palm, fingertip, or index direction) targets distant UI elements. Users perform a gesture ( | For distant objects beyond arm's reach, a virtual ray (from palm, fingertip, or index direction) targets distant UI elements. Users perform a gesture (for example pinch) to activate or select the targeted item. This allows interaction with objects throughout the virtual environment without physical reach limitations. | ||
=== Multimodal Interaction === | === Multimodal Interaction === | ||
| Line 188: | Line 188: | ||
* '''System UI & Productivity''': Controller-free navigation, window management, and typing/pointing surrogates in spatial desktops. Natural file manipulation, multitasking across virtual screens, and interface control without handheld devices.<ref name="AppleGestures" /> | * '''System UI & Productivity''': Controller-free navigation, window management, and typing/pointing surrogates in spatial desktops. Natural file manipulation, multitasking across virtual screens, and interface control without handheld devices.<ref name="AppleGestures" /> | ||
* '''Gaming & Entertainment''': Titles such as ''Hand Physics Lab'' showcase free-hand puzzles and physics interactions using optical hand tracking on Quest.<ref name="HPL_RoadToVR" /> Games and creative applications use hand interactions, | * '''Gaming & Entertainment''': Titles such as ''Hand Physics Lab'' showcase free-hand puzzles and physics interactions using optical hand tracking on Quest.<ref name="HPL_RoadToVR" /> Games and creative applications use hand interactions, for example a puzzle game might let the player literally reach out and grab puzzle pieces in VR, or users can play virtual piano or create pottery simulations. | ||
* '''Training & Simulation''': Natural hand use improves ecological validity for assembly, maintenance, and surgical rehearsal in enterprise, medical, and industrial contexts.<ref name="Frontiers2021" /> Workers can practice complex procedures in safe virtual environments, developing muscle memory that transfers to real-world tasks. | * '''Training & Simulation''': Natural hand use improves ecological validity for assembly, maintenance, and surgical rehearsal in enterprise, medical, and industrial contexts.<ref name="Frontiers2021" /> Workers can practice complex procedures in safe virtual environments, developing muscle memory that transfers to real-world tasks. | ||
| Line 209: | Line 209: | ||
* On '''[[Microsoft HoloLens 2]]''', a 2024 study comparing to a Vicon motion-capture reference found millimeter-scale fingertip errors (approximately 2-4 mm) in a tracing task, with good agreement for pinch span and many grasping joint angles.<ref name="HL2Accuracy" /> | * On '''[[Microsoft HoloLens 2]]''', a 2024 study comparing to a Vicon motion-capture reference found millimeter-scale fingertip errors (approximately 2-4 mm) in a tracing task, with good agreement for pinch span and many grasping joint angles.<ref name="HL2Accuracy" /> | ||
Real-world performance also depends on lighting, hand pose, occlusions ( | Real-world performance also depends on lighting, hand pose, occlusions (for example fingers hidden by other fingers), camera field of view, and motion speed. Runtime predictors reduce jitter and tracking loss but cannot eliminate these effects entirely.<ref name="Frontiers2021" /><ref name="MetaHands21" /> | ||
== Advantages == | == Advantages == | ||
| Line 220: | Line 220: | ||
* '''Expressiveness''': Hands allow a wide range of gesture expressions. In contrast to a limited set of controller buttons, hand tracking can capture nuanced movements. This enables richer interactions (such as sculpting a 3D model with complex hand movements) and communication (subtle social gestures, sign language, etc.). Important for social presence, waving, pointing, subtle finger cues enhance non-verbal communication. | * '''Expressiveness''': Hands allow a wide range of gesture expressions. In contrast to a limited set of controller buttons, hand tracking can capture nuanced movements. This enables richer interactions (such as sculpting a 3D model with complex hand movements) and communication (subtle social gestures, sign language, etc.). Important for social presence, waving, pointing, subtle finger cues enhance non-verbal communication. | ||
* '''Hygiene & Convenience''': Especially in public or shared XR setups, hand tracking can be advantageous since users do not need to touch common surfaces or devices. Touchless interfaces have gained appeal for reducing contact points. Moreover, not having to pick up or hold hardware means quicker setup and freedom to use one's hands spontaneously ( | * '''Hygiene & Convenience''': Especially in public or shared XR setups, hand tracking can be advantageous since users do not need to touch common surfaces or devices. Touchless interfaces have gained appeal for reducing contact points. Moreover, not having to pick up or hold hardware means quicker setup and freedom to use one's hands spontaneously (for example switching between real objects and virtual interface by just moving hands). No shared controllers required; quicker task switching between physical tools and virtual UI. | ||
== Challenges and Limitations == | == Challenges and Limitations == | ||
| Line 226: | Line 226: | ||
=== Technical Limitations === | === Technical Limitations === | ||
* '''Occlusion & Field of View''': Self-occluding poses ( | * '''Occlusion & Field of View''': Self-occluding poses (for example fists, crossed fingers) and hands leaving camera FOV can cause tracking loss. Predictive tracking mitigates but cannot remove this. Ensuring that hand tracking works in all conditions is difficult. Optical systems can struggle with poor lighting, motion blur from fast hand movements, or when the hands leave the camera's field of view (for example reaching behind one's back). Even depth cameras have trouble if the sensors are occluded or if reflective surfaces confuse the measurements.<ref name="Frontiers2021" /><ref name="MediaPipeHands" /> | ||
* '''Latency & Fast Motion''': Even 70 ms delay can feel disconnected. Fast motion burdens mobile compute. Continuous updates ( | * '''Latency & Fast Motion''': Even 70 ms delay can feel disconnected. Fast motion burdens mobile compute. Continuous updates (for example Quest "Hands 2.x") have narrowed gaps to controllers but not eliminated them. There can also be a slight latency in hand tracking responses due to processing, which, if not minimized, can affect user performance.<ref name="MetaHands22" /> | ||
* '''Lighting & Reflectance Sensitivity''': Purely optical methods remain sensitive to extreme lighting conditions and reflective surfaces, though IR illumination helps.<ref name="Frontiers2021" /> | * '''Lighting & Reflectance Sensitivity''': Purely optical methods remain sensitive to extreme lighting conditions and reflective surfaces, though IR illumination helps.<ref name="Frontiers2021" /> | ||