Hand tracking: Difference between revisions
Xinreality (talk | contribs) |
Xinreality (talk | contribs) No edit summary |
||
| Line 38: | Line 38: | ||
! Device Name !! Year !! Key Individuals/Organizations !! Core Technology !! Historical Significance | ! Device Name !! Year !! Key Individuals/Organizations !! Core Technology !! Historical Significance | ||
|- | |- | ||
| '''Sayre Glove''' || 1977 || Daniel Sandin, Thomas DeFanti (University of Illinois) || Light-based sensors using flexible tubes and photocells to measure light attenuation || First data glove prototype; established finger flexion measurement for computer input | | '''[[Sayre Glove]]''' || 1977 || Daniel Sandin, Thomas DeFanti (University of Illinois) || Light-based sensors using flexible tubes and photocells to measure light attenuation || First data glove prototype; established finger flexion measurement for computer input | ||
|- | |- | ||
| '''Digital Data Entry Glove''' || 1983 || Gary Grimes (Bell Labs) || Multi-sensor system with flex, tactile, and inertial sensors || Pioneered integration of multiple sensor types for complex gesture recognition | | '''[[Digital Data Entry Glove]]''' || 1983 || Gary Grimes (Bell Labs) || Multi-sensor system with flex, tactile, and inertial sensors || Pioneered integration of multiple sensor types for complex gesture recognition | ||
|- | |- | ||
| '''VPL DataGlove''' || 1987 || Thomas Zimmerman, Jaron Lanier (VPL Research) || Fiber optic cables with LEDs and photosensors || First commercially successful data glove; iconic early VR technology | | '''[[VPL DataGlove]]''' || 1987 || Thomas Zimmerman, Jaron Lanier (VPL Research) || Fiber optic cables with LEDs and photosensors || First commercially successful data glove; iconic early VR technology | ||
|- | |- | ||
| '''Power Glove''' || 1989 || Mattel (licensed from VPL) || Low-cost resistive ink flex sensors and ultrasonic tracking || First affordable mass-market data glove for consumers | | '''[[Power Glove]]''' || 1989 || Mattel (licensed from VPL) || Low-cost resistive ink flex sensors and ultrasonic tracking || First affordable mass-market data glove for consumers | ||
|- | |- | ||
| '''CyberGlove''' || Early 1990s || Virtual Technologies, Inc. || Thin foil strain gauges measuring up to 22 joint angles || High-precision glove for research and professional applications | | '''[[CyberGlove]]''' || Early 1990s || Virtual Technologies, Inc. || Thin foil strain gauges measuring up to 22 joint angles || High-precision glove for research and professional applications | ||
|} | |} | ||
| Line 55: | Line 55: | ||
=== 2010s: Optical Tracking and Controller-Free Era === | === 2010s: Optical Tracking and Controller-Free Era === | ||
A pivotal shift occurred in 2010 with the founding of '''Leap Motion''' (later [[Ultraleap]]) by Michael Buckwald, David Holz, and John Gibb, who aimed to create affordable, high-precision optical hand tracking. In 2010, Microsoft also released the '''[[Kinect]]''' sensor for Xbox 360, which popularized the use of depth cameras for full-body and hand skeleton tracking in gaming and research.<ref name="Kinect" /> | A pivotal shift occurred in 2010 with the founding of '''[[Leap Motion]]''' (later [[Ultraleap]]) by Michael Buckwald, David Holz, and John Gibb, who aimed to create affordable, high-precision optical hand tracking. In 2010, Microsoft also released the '''[[Kinect]]''' sensor for Xbox 360, which popularized the use of depth cameras for full-body and hand skeleton tracking in gaming and research.<ref name="Kinect" /> | ||
The '''Leap Motion Controller''', released in 2013, was a small USB device with stereo IR cameras that could track both hands with fine precision (down to approximately 0.01 mm according to specifications) in a limited space above the device.<ref name="LeapWiki" /><ref name="LeapWikipedia" /> This device revolutionized VR by enabling untethered, gesture-based input. Developers and enthusiasts mounted Leap Motion sensors onto VR headsets to experiment with hand input in VR, spurring further interest in the technique. Their flagship controller used two IR cameras and three IR LEDs to track hands at up to 200 Hz over a 60 cm × 60 cm interactive zone. | The '''[[Leap Motion Controller]]''', released in 2013, was a small USB device with stereo IR cameras that could track both hands with fine precision (down to approximately 0.01 mm according to specifications) in a limited space above the device.<ref name="LeapWiki" /><ref name="LeapWikipedia" /> This device revolutionized VR by enabling untethered, gesture-based input. Developers and enthusiasts mounted Leap Motion sensors onto VR headsets to experiment with hand input in VR, spurring further interest in the technique. Their flagship controller used two IR cameras and three IR LEDs to track hands at up to 200 Hz over a 60 cm × 60 cm interactive zone. | ||
In 2016, Leap Motion's '''Orion''' software update improved robustness against occlusion and lighting variations, boosting adoption in VR development.<ref name="Orion" /> The company was acquired by Ultrahaptics in 2019, rebranding as [[Ultraleap]] and expanding into mid-air haptics.<ref name="UltrahapticsAcq" /> | In 2016, Leap Motion's '''[[Orion]]''' software update improved robustness against occlusion and lighting variations, boosting adoption in VR development.<ref name="Orion" /> The company was acquired by Ultrahaptics in 2019, rebranding as [[Ultraleap]] and expanding into mid-air haptics.<ref name="UltrahapticsAcq" /> | ||
On the AR side, '''Microsoft HoloLens''' (first version, 2016) included simple gesture input such as "air tap" and "bloom" using camera-based hand recognition, though with limited gestures rather than full tracking. | On the AR side, '''[[Microsoft HoloLens]]''' (first version, 2016) included simple gesture input such as "air tap" and "bloom" using camera-based hand recognition, though with limited gestures rather than full tracking. | ||
By the late 2010s, [[inside-out tracking]] cameras became standard in new VR and AR hardware, and companies began leveraging them for hand tracking. The '''[[Oculus Quest]]''', a standalone VR headset released in 2019, initially launched with traditional controller input. At Oculus Connect 6 in September 2019, hand tracking was announced, and in late 2019 an experimental update introduced controller-free hand tracking using its built-in cameras.<ref name="Meta2019" /><ref name="SpectreXR2022" /> This made the Quest one of the first mainstream VR devices to offer native hand tracking to consumers, showcasing surprisingly robust performance, albeit with some limitations in fast motion and certain angles. This inside-out system used the headset's monochrome cameras and AI for controller-free interactions, marking the mainstream consumer debut. | By the late 2010s, [[inside-out tracking]] cameras became standard in new VR and AR hardware, and companies began leveraging them for hand tracking. The '''[[Oculus Quest]]''', a standalone VR headset released in 2019, initially launched with traditional controller input. At Oculus Connect 6 in September 2019, hand tracking was announced, and in late 2019 an experimental update introduced controller-free hand tracking using its built-in cameras.<ref name="Meta2019" /><ref name="SpectreXR2022" /> This made the Quest one of the first mainstream VR devices to offer native hand tracking to consumers, showcasing surprisingly robust performance, albeit with some limitations in fast motion and certain angles. This inside-out system used the headset's monochrome cameras and AI for controller-free interactions, marking the mainstream consumer debut. | ||
The '''Microsoft HoloLens 2''' (2019) greatly expanded hand tracking capabilities with fully articulated tracking, allowing users to touch and grasp virtual elements directly. The system tracked 25 points of articulation per hand, demonstrating the benefit of more natural interactions for enterprise AR use cases and eliminating the limited "air tap" gestures of its predecessor.<ref name="Develop3D2019" /><ref name="DirectManipulation" /> | The '''[[Microsoft HoloLens 2]]''' (2019) greatly expanded hand tracking capabilities with fully articulated tracking, allowing users to touch and grasp virtual elements directly. The system tracked 25 points of articulation per hand, demonstrating the benefit of more natural interactions for enterprise AR use cases and eliminating the limited "air tap" gestures of its predecessor.<ref name="Develop3D2019" /><ref name="DirectManipulation" /> | ||
=== 2020s: AI-Driven Refinements and Mainstream Integration === | === 2020s: AI-Driven Refinements and Mainstream Integration === | ||
The 2020s brought AI enhancements and broader integration. Meta's '''MEgATrack''' (2020), deployed on Quest, used four fisheye monochrome cameras and neural networks for 60 Hz PC tracking and 30 Hz mobile, with low jitter and large working volumes.<ref name="MEgATrack" /> | The 2020s brought AI enhancements and broader integration. Meta's '''[[MEgATrack]]''' (2020), deployed on Quest, used four fisheye monochrome cameras and neural networks for 60 Hz PC tracking and 30 Hz mobile, with low jitter and large working volumes.<ref name="MEgATrack" /> | ||
Ultraleap's '''Gemini''' software update (2021) represented a major overhaul with stronger two-hand interactions and initialization speed for stereo-IR modules and integrated OEM headsets.<ref name="Gemini" /> | Ultraleap's '''Gemini''' software update (2021) represented a major overhaul with stronger two-hand interactions and initialization speed for stereo-IR modules and integrated OEM headsets.<ref name="Gemini" /> | ||
Meta continued improving the feature over time with successive updates. The '''Hands 2.1''' update (2022) and '''Hands 2.2''' update (2023) reduced apparent latency and improved fast-motion handling and recovery after tracking loss.<ref name="MetaHands21" /><ref name="MetaHands22" /> Subsequent devices like the '''[[Meta Quest Pro]]''' (2022) and '''[[Meta Quest 3]]''' (2023) included more advanced camera systems and neural processing to further refine hand tracking. | Meta continued improving the feature over time with successive updates. The '''[[Hands 2.1]]''' update (2022) and '''[[Hands 2.2]]''' update (2023) reduced apparent latency and improved fast-motion handling and recovery after tracking loss.<ref name="MetaHands21" /><ref name="MetaHands22" /> Subsequent devices like the '''[[Meta Quest Pro]]''' (2022) and '''[[Meta Quest 3]]''' (2023) included more advanced camera systems and neural processing to further refine hand tracking. | ||
In the 2020s, hand tracking became an expected feature in many XR devices. An analysis by SpectreXR noted that the percentage of new VR devices supporting hand tracking jumped from around 21% in 2021 to 46% in 2022, as more manufacturers integrated the technology.<ref name="SpectreXR2023" /> At the same time, the cost barrier dropped dramatically, with the average price of hand-tracking-capable VR headsets falling by approximately 93% from 2021 to 2022, making the technology far more accessible.<ref name="SpectreXR2023" /> | In the 2020s, hand tracking became an expected feature in many XR devices. An analysis by SpectreXR noted that the percentage of new VR devices supporting hand tracking jumped from around 21% in 2021 to 46% in 2022, as more manufacturers integrated the technology.<ref name="SpectreXR2023" /> At the same time, the cost barrier dropped dramatically, with the average price of hand-tracking-capable VR headsets falling by approximately 93% from 2021 to 2022, making the technology far more accessible.<ref name="SpectreXR2023" /> | ||
| Line 85: | Line 85: | ||
! Year !! Development / System !! Notes | ! Year !! Development / System !! Notes | ||
|- | |- | ||
| 1977 || '''Sayre Glove''' || First instrumented glove for virtual interaction using light sensors and photocells | | 1977 || '''[[Sayre Glove]]''' || First instrumented glove for virtual interaction using light sensors and photocells | ||
|- | |- | ||
| 1987 || '''VPL DataGlove''' || First commercial data glove measuring finger bend and hand orientation | | 1987 || '''[[VPL DataGlove]]''' || First commercial data glove measuring finger bend and hand orientation | ||
|- | |- | ||
| 2010 || '''Microsoft Kinect''' || Popularized full-body and hand tracking using IR depth camera for gaming | | 2010 || '''[[Microsoft Kinect''' || Popularized full-body and hand tracking using IR depth camera for gaming | ||
|- | |- | ||
| 2013 || '''Leap Motion Controller''' || Small USB peripheral with dual infrared cameras for high-precision hand tracking | | 2013 || '''[[Leap Motion Controller]]''' || Small USB peripheral with dual infrared cameras for high-precision hand tracking | ||
|- | |- | ||
| 2016 || '''HoloLens (1st gen)''' || AR headset with gesture input (air tap and bloom) | | 2016 || '''[[HoloLens]] (1st gen)''' || AR headset with gesture input (air tap and bloom) | ||
|- | |- | ||
| 2019 || '''Oculus/Meta Quest (update)''' || First mainstream VR device with fully integrated controller-free hand tracking | | 2019 || '''[[Oculus]]/[[Meta Quest]] (update)''' || First mainstream VR device with fully integrated controller-free hand tracking | ||
|- | |- | ||
| 2019 || '''HoloLens 2''' || Fully articulated hand tracking (25 points per hand) for direct manipulation | | 2019 || '''[[HoloLens 2]]''' || Fully articulated hand tracking (25 points per hand) for direct manipulation | ||
|- | |- | ||
| 2022 || '''Meta Quest Pro''' || Improved cameras and dedicated computer vision co-processor for responsive hand tracking | | 2022 || '''[[Meta Quest Pro]]''' || Improved cameras and dedicated computer vision co-processor for responsive hand tracking | ||
|- | |- | ||
| 2023 || '''Apple Vision Pro''' || Mixed reality device using hand and eye tracking as primary input, eliminating controllers | | 2023 || '''[[Apple Vision Pro]]''' || Mixed reality device using hand and eye tracking as primary input, eliminating controllers | ||
|- | |- | ||
| 2023 || '''Meta Quest 3''' || Enhanced hand tracking with improved latency and accuracy | | 2023 || '''[[Meta Quest 3]]''' || Enhanced hand tracking with improved latency and accuracy | ||
|} | |} | ||
== Technology and Implementation == | == Technology and Implementation == | ||
Most contemporary implementations are '''markerless''' (no gloves or markers required), relying on cameras and sensors to detect hand motions rather than requiring wearables. They estimate hand pose directly from one or more cameras using [[computer vision]] and [[neural network]]s, optionally fused with active depth sensing.<ref name="Frontiers2021" /> | Most contemporary implementations are '''[[markerless]]''' (no gloves or markers required), relying on cameras and sensors to detect hand motions rather than requiring wearables. They estimate hand pose directly from one or more cameras using [[computer vision]] and [[neural network]]s, optionally fused with active depth sensing.<ref name="Frontiers2021" /> | ||
=== Optical and AI Pipeline === | === Optical and AI Pipeline === | ||
| Line 134: | Line 134: | ||
=== Instrumented Gloves === | === Instrumented Gloves === | ||
An alternative approach to camera-based tracking is the use of wired gloves or other wearables that directly capture finger movements. This was the earliest form of hand tracking in VR history. Modern glove-based controllers (often with [[haptic feedback]]) still exist, such as the '''HaptX''' gloves which combine precise motion tracking with force-feedback to simulate touch, or products like Manus VR gloves. These devices can provide very accurate finger tracking and even capture nuanced motions (like pressure or stretch) that camera systems might miss, but they require the user to wear hardware on their hands, which can be less convenient. Glove-based tracking is often used in enterprise and training applications or research, where maximum precision and feedback is needed. | An alternative approach to camera-based tracking is the use of wired gloves or other wearables that directly capture finger movements. This was the earliest form of hand tracking in VR history. Modern glove-based controllers (often with [[haptic feedback]]) still exist, such as the '''[[HaptX]]''' gloves which combine precise motion tracking with force-feedback to simulate touch, or products like Manus VR gloves. These devices can provide very accurate finger tracking and even capture nuanced motions (like pressure or stretch) that camera systems might miss, but they require the user to wear hardware on their hands, which can be less convenient. Glove-based tracking is often used in enterprise and training applications or research, where maximum precision and feedback is needed. | ||
=== Development Standards === | === Development Standards === | ||
| Line 168: | Line 168: | ||
This is prominent in [[Microsoft HoloLens 2]] design guidance for natural, tactile mental models.<ref name="DirectManipulation" /> In VR training and simulation, trainees can practice assembling a device by virtually grabbing parts and fitting them together with their hands. This direct manipulation is more immersive than using a laser pointer or controller buttons. | This is prominent in [[Microsoft HoloLens 2]] design guidance for natural, tactile mental models.<ref name="DirectManipulation" /> In VR training and simulation, trainees can practice assembling a device by virtually grabbing parts and fitting them together with their hands. This direct manipulation is more immersive than using a laser pointer or controller buttons. | ||
=== Ray-Based Selection (Indirect Interaction) === | === [[Ray-Based Selection]] (Indirect Interaction) === | ||
For distant objects beyond arm's reach, a virtual ray (from palm, fingertip, or index direction) targets distant UI elements. Users perform a gesture (e.g., pinch) to activate or select the targeted item. This allows interaction with objects throughout the virtual environment without physical reach limitations. | For distant objects beyond arm's reach, a virtual ray (from palm, fingertip, or index direction) targets distant UI elements. Users perform a gesture (e.g., pinch) to activate or select the targeted item. This allows interaction with objects throughout the virtual environment without physical reach limitations. | ||
=== Multimodal Interaction === | === Multimodal Interaction === | ||
Combining hand tracking with other inputs enhances interaction: | Combining hand tracking with other inputs enhances interaction: | ||
* '''Gaze-and-pinch''' (Apple Vision Pro): [[Eye tracking]] rapidly targets UI elements, while a subtle pinch gesture confirms selection. This is the primary paradigm on [[Apple Vision Pro]], allowing control without holding up hands constantly—a brief pinch at waist level suffices.<ref name="AppleGestures" /><ref name="UploadVR2023" /> | * '''[[Gaze-and-pinch]]''' ([[Apple Vision Pro]]): [[Eye tracking]] rapidly targets UI elements, while a subtle pinch gesture confirms selection. This is the primary paradigm on [[Apple Vision Pro]], allowing control without holding up hands constantly—a brief pinch at waist level suffices.<ref name="AppleGestures" /><ref name="UploadVR2023" /> | ||
* '''Voice and gesture''': Verbal commands with hand confirmation | * '''[[Voice]] and [[gesture]]''': Verbal commands with hand confirmation | ||
* '''Hybrid controller/hands''': Seamless switching between modalities | * '''Hybrid controller/hands''': Seamless switching between modalities | ||
| Line 254: | Line 254: | ||
=== Advanced Haptics === | === Advanced Haptics === | ||
Research directions to address the lack of tactile feedback include: | Research directions to address the lack of tactile feedback include: | ||
* Ultrasonic mid-air haptics (Ultraleap) | * Ultrasonic mid-air haptics ([[Ultraleap]]) | ||
* Haptic gloves with force feedback (HaptX, Manus VR) | * Haptic gloves with force feedback ([[HaptX]],[[Manus VR]]) | ||
* Electrical muscle stimulation | * Electrical muscle stimulation | ||
* Skin-integrated haptic patches | * Skin-integrated haptic patches | ||