Hand tracking
More actions
Hand tracking estimates the position, orientation or articulated configuration of a person's hands for digital input and motion capture. In virtual reality (VR), augmented reality (AR) and mixed reality, the resulting hand representation can support touching, grabbing, pointing and gesturing. Optical systems use cameras and a hand model to estimate joint locations, including some hidden parts. Instrumented data gloves instead measure quantities such as finger bending; an additional tracker may locate the glove in space.[1][2]
Tracking and gesture recognition are related but distinct. Tracking supplies an evolving pose; interaction software interprets it as a pinch, grab or other command. Applications also decide how to show a hand, how to select a target and how to recover when tracking disappears. Using the hands can support expression and familiar actions, but does not itself establish better task performance, physical contact or accessibility for every user.[3]
History
Early developments (1970s-1990s)
Instrumented gloves preceded modern camera-based XR input. The Sayre glove is an early example documented in a 1977 Electronic Visualization Laboratory project report. The report credits Rich Sayre's idea and describes a light source and photocell at opposite ends of a flexible tube. Bending or compressing the tube reduces transmitted light, producing a control signal. Its construction allowed adjustment of the glove and sensor positions to the wearer. This mechanism should not be conflated with a later fiber-optic bend sensor.[4] An academic history of computer graphics also identifies earlier typing-glove work by Uttal in 1962, so the Sayre device is better described as an early instrumented glove than as the first glove input device.[5]
Gary Grimes's Digital data entry glove interface device, assigned to Bell Telephone Laboratories and granted as US4414537A in 1983, combined finger bend, contact, tilt and motion sensing. An embodiment encoded a single-hand manual alphabet into data-entry characters. This was a glove interface for defined hand configurations, rather than a demonstration of unrestricted sign-language interpretation.[6]
Thomas Zimmerman's Optical flex sensor patent was filed on September 29, 1982 and granted on September 17, 1985 as US4542291A. The distinction matters when dating the development: 1982 is the application date.[7] Later VPL DataGlove systems used optical finger-bend sensing with separate hand-position tracking. The historical account describes fitting and recalibration issues, including effects of hand size and placement.[5]
The Powerhouse museum's catalog places the VPL DataGlove around 1987 and the Mattel Power Glove in 1989. The Power Glove was an input accessory for the Nintendo Entertainment System manufactured under license from Abrams-Gentile Entertainment. It used resistive ink to sense bending in four fingers and ultrasonic transmitters with a receiver assembly containing three microphones to locate the hand. This arrangement differs from both an optical DataGlove and a modern camera-estimated skeleton.[8]
| Early example | Date or documentary basis | Sensing and scope |
|---|---|---|
| Sayre glove | 1977 project report | Flexible optical tubes generated bend-dependent control signals; adjustable construction.[4] |
| Grimes data-entry glove | Patent granted 1983 | Defined configurations sensed with bend, contact, tilt and motion elements.[6] |
| Zimmerman optical flex sensor | Filed 1982; granted 1985 | Patent for an optical bend sensor, distinct from the date of a later commercial glove.[7] |
| VPL DataGlove | Circa 1987 in the museum account | Optical finger-bend glove lineage.[8] |
| Mattel Power Glove | 1989 | Finger-bend input and ultrasonic hand-location system for NES.[8] |
2000s: instrumented sensing and movement research
Ingram and colleagues' 2008 study, The statistics of natural hand movements, used a CyberGlove to record 19 joint angles at 84 Hz during daily activities. Six healthy male participants supplied about 17 hours of recordings. The glove was calibrated for each person and the researchers checked alignment before sessions. Finger movements were correlated, with the thumb and index finger relatively independent in these observations. The study illustrates both the usefulness of joint-angle recording and the need to state the measured joints and population; it does not establish a universal number of independently measurable hand motions.[9]
Commercial glove configurations also differ. CyberGlove II documentation describes versions with 18 or 22 resistive bend sensors and optional external position/orientation trackers.[2] The 5DT Data Glove Ultra has five- and fourteen-sensor variants; the fourteen-sensor version measures two flexure locations per finger and finger abduction. These are product-specific measurement layouts, rather than interchangeable anatomical skeletons.[10]
2010s: optical tracking and controller-free input
The Leap Motion Controller was an early dedicated optical hand-input device examined by researchers in 2013. Weichert and colleagues tested a preliminary developer unit using a robot-guided reference pen. Its hardware used two infrared cameras and three infrared emitters. Their controlled tests reported static deviations below 0.2 mm and average dynamic deviations around 1.2 mm; they could not substantiate a 0.01 mm accuracy claim. The test measured a tracked pen, not the anatomical positions of all joints in a moving hand.[11]
Research also developed articulated pose estimation from depth images. Qian and colleagues' 2014 system represented a hand with spheres and fitted the model to a depth point cloud. Hybrid gradient-based and stochastic optimization, together with finger detection for initialization, addressed local minima and recovery from tracking failure. It is an example of model fitting rather than recognition of only a few discrete gestures.[12]
On December 9, 2019, Oculus announced an early hand-tracking feature for the original Oculus Quest, scheduled for the v12 rollout that week, with the developer SDK following the next week. The announcement described using the headset's existing monochrome cameras and model-based hand inference, without adding an active depth sensor or processor. The early feature complemented controller input.[13] Leap Motion and Ultrahaptics also combined as Ultraleap in 2019.[14]
2020s: model refinement and platform integration
The 2020 MEgATrack paper described articulated tracking from four monochrome fisheye cameras on a headset. The pipeline combined image detections, a hand model and tracking history, with reported operation at 60 Hz on a PC and 30 Hz on mobile hardware in that implementation. The paper also documented failures with uncommon viewpoints and severe hand-object occlusion. This original research describes a particular system, rather than every subsequent Quest implementation.[15]
Ultraleap's Gemini V5.2 migration documentation records changes to the tracking engine and integration requirements. Older Unity, Unreal and OpenXR integration tools required updates; installing multiple tracking-service versions could produce conflicts. These version boundaries distinguish an engine update from a guarantee that an old application works with every later tracking package.[16]
Meta's January 19, 2023 Hand Tracking 2.1 announcement associated improvements with Quest OS v47, including faster recovery, less jitter and better prediction and system pinch behavior.[17] Its August 21, 2023 Hand Tracking 2.2 announcement for v56 claimed latency reductions relative to 2.1 of up to 40% during typical use and up to 75% during fast movements, with an optional Fast Motion Mode. These are dated vendor comparisons, not independently measured absolute latency values.[18] The February 2023 v50 Direct Touch announcement described pressing buttons, scrolling and using a virtual keyboard with the fingers.[19]
Apple's WWDC23 ARKit presentation explained skeletal hand data for immersive applications on Apple Vision Pro, with a HandTrackingProvider and permission-controlled access.[20] Apple user guidance combines eye targeting with finger pinches, taps and drags. Visible hands can rest on a lap or table.[21]
| Milestone | What it establishes |
|---|---|
| 2013 Leap Motion research | Dedicated optical hardware was evaluated in a controlled pen-tracking experiment.[11] |
| December 2019 Quest announcement | A consumer headset added an early hand-input feature using its existing cameras.[13] |
| 2020 MEgATrack paper | An original multi-camera articulated tracking pipeline and its reported limitations.[15] |
| 2023 Quest software announcements | Dated improvements to tracking and direct-touch interaction, with configuration-specific claims.[17][19] |
| 2023 ARKit presentation | An application API for permission-controlled skeletal hand data in immersive spaces.[20] |
| April 2024 OpenXR 1.1 | Promotion of a grip-surface action pose; articulated joint tracking remained an extension.[22] |
Technology and implementation
Optical and AI pipeline
An optical tracker must find the hand, infer its articulation and maintain an estimate as it moves. These stages can share a model, but detecting a hand in an image does not by itself recover all finger positions. The 2020 MediaPipe Hands pipeline illustrates a two-stage approach: a detector finds an oriented palm region and a landmark network estimates 21 landmarks within the crop. Its original output included image coordinates and depth relative to the wrist. Tracking reused the preceding frame's crop, invoking detection again when the hand-presence estimate indicated a loss. Those relative-depth landmarks are not, by themselves, a calibrated headset-space measurement.[23]
Google's current Hand Landmarker guide retains the distinction between palm detection, hand-presence confidence and tracking confidence. In video and live-stream modes, preceding landmarks can supply a region for the next frame; inadequate presence or tracking triggers detection again. Its model combines real-image and synthetic training examples. A detector's inference time, landmark count or presence score should not be substituted for the latency and spatial accuracy of a complete XR application.[24]
Model constraints also fill gaps in image evidence. Ultraleap describes using a model of the hand's underlying joint and bone structure to reconstruct fingers when parts are hidden.[1] MEgATrack similarly uses tracking history and geometric constraints, but its evaluation explicitly notes severe object-occlusion failures and incomplete labels for some difficult occluded frames. A plausible rendered hand can therefore include inferred joints, and a benchmark with missing labels can leave important failures unevaluated.[15]
An API skeleton is a representation chosen for software. The OpenXR default hand-joint set has 26 entries: palm, wrist, four thumb entries and five for each other digit.[25] That count should not be called 26 or 27 anatomical degrees of freedom. Joint positions, rotations, digit flexion and overall hand location describe different quantities.[9]
Inside-out versus outside-in tracking
A headset-mounted optical sensor observes hands from the wearer's moving viewpoint. A desktop or externally mounted camera observes a region defined by its installation. This distinction affects camera-to-display alignment and the directions in which a hand can obscure another finger. Ultraleap's integration guidance distinguishes desktop and head-mounted tracking configurations and discusses offsets between the sensor and the user's eyes. Correctly applying that mounting relationship is necessary to display the estimated hand in the intended place.[26]
The application also needs a coordinate reference. OpenXR's xrLocateHandJointsEXT locates an array of joints relative to a requested base space at a specified time. Comparing its result with an external reference therefore requires a shared spatial relationship and corresponding times; coordinates from different origins cannot be treated as directly comparable measurements.[27]
Tracking data can remain valid even when some of it is inferred. Under the OpenXR hand-tracking contract, an active hand supplies valid joint positions and orientations, but individual tracked flags can be unset. When the tracker is inactive because no hand input was detected or the application lost input focus, the joint validity flags are unset. Software should use that state rather than continue acting on an old visible hand.[28]
Depth sensing and infrared
Depth input, stereo infrared images and ordinary RGB images provide different observations. Qian's depth-based prototype fitted a sphere model to a point cloud; it evaluated five fingertips and the wrist against manually labeled frames from six subjects, with subject-specific overall hand scale. Its measurements and desktop processing rate characterize that research system. They do not establish the range or error of a current headset.[12]
Ultraleap's dedicated cameras use stereo infrared imaging and infrared illumination. The software reconstructs the hand using the images and its model; a camera is not a direct sensor of every internal finger joint.[1] The original Quest hand-tracking announcement instead described use of the headset's existing monochrome cameras without an added active depth sensor.[13] MediaPipe's original research used a single RGB camera and relative-depth estimation.[23] These examples show why an XR headset's general depth or scene-sensing features should not automatically be identified as its hand-tracking mechanism.
Instrumented gloves
Glove sensors measure quantities determined by their placement. CyberGlove II bend sensors cover finger flexion, abduction and other hand/wrist articulation, while its documented optional external trackers supply overall position and orientation. A glove with finger-angle data alone is not a complete spatial hand tracker.[2] The five- and fourteen-sensor 5DT variants likewise provide different flexure/abduction coverage, so applications need to match their model to the selected glove.[10]
Calibration maps sensor readings to a wearer and joint representation, as illustrated by Ingram's individually calibrated glove recordings.[9] Historical optical gloves also required fitting and recalibration; sensor slippage and a poor fit could change the mapping.[5]
A haptic glove may combine tracking with a separate feedback subsystem. HaptX's 2024 G1 specification sheet separately lists motion capture, tactile actuators, force actuators and an airpack. Its resolution, update rate and feedback response figures describe different subsystems. In particular, a listed positional resolution is not a measured anatomical accuracy, and haptic engagement time is not identical to tracking latency.[29]
Development standards
OpenXR exposes articulated hands through XR_EXT_hand_tracking. Applications must check extension/runtime capability, including supportsHandTracking, and create an appropriate hand tracker. Joint locations can include radii, with optional joint-velocity output. Supporting OpenXR does not, by itself, mean that a runtime supports articulated hand tracking.[30]
OpenXR 1.1 promoted the functionality of XR_EXT_palm_pose as the grip_surface pose identifier. This is an action pose for anchoring content relative to a hand, including a hand position inferred from a controller. It is separate from the 26-joint skeletal interface and did not promote XR_EXT_hand_tracking into core.[22] The joint API also permits emulation of an unobstructed hand configuration when available input is limited or a held controller constrains the physical hand. An application cannot interpret every returned pose as a directly observed anatomical measurement.[28]
The WebXR Hand Input Module defines a different schema with 25 joint spaces, including the wrist and finger entries but no separate palm entry. The June 5, 2024 document is a W3C Working Draft. Applications request the hand-tracking session feature; an input source's hand remains null without it. Its rules allow hidden joints to be emulated and require all joints to become unavailable together when joint poses cannot be supplied. Thus OpenXR, WebXR and model-specific landmark indices require an explicit mapping rather than an assumed identical array.[31]
Engine libraries add representation and interaction behavior above these interfaces. The archived MRTK2 guide exposes joint poses through queries such as TryGetJoint, which can fail when a joint is unavailable. It warns that joint GameObjects can be destroyed when tracking is lost, and that complex joint prefabs or hand-mesh rendering incur performance costs. This is a version-specific implementation example; simulated hands in an editor do not validate physical-device tracking.[32]
Apple's 2023 ARKit presentation describes left/right hand anchors rooted at the wrist, joint transforms and tracked status. Applications running in a Full Space could request the HandTrackingProvider with user authorization. This skeletal-data interface is distinct from receiving a normal system-mediated selection gesture.[20] Current Meta documentation similarly distinguishes the tracking data supplied by its core integration from higher-level interaction components, and describes Meta Link hands in a Windows Unity Editor development workflow. That workflow should not be read as general hand-input support in every PC VR application.[33]
Notable platforms
The following comparison separates a documented input architecture from the application interface. Software behavior and permissions vary by device, runtime and release; the cited documentation was checked on October 13, 2026. Historical test results belong to the hardware and software configuration in each experiment.
| Platform or integration | Documented hand input | Application and configuration boundary |
|---|---|---|
| Oculus Quest (original feature) | Existing monochrome headset cameras used for model-based hand inference. | December 2019 early rollout; no added active depth sensor required for that feature.[13] |
| Quest 2, Quest Pro, Quest 3 and Quest 3S | Current Meta Multimodal documentation lists concurrent hand and controller tracking for these models. | Dedicated October 7, 2026 documentation lists interactions with other features and limitations when detecting controllers held in a hand. Support must be checked against the selected software configuration.[34] |
| HoloLens 2 | Near touch, grab and articulated-hand input are documented in the platform's interaction guidance. | Original HoloLens does not provide the same direct-manipulation model; archived MRTK2 joint queries can fail and lost-hand objects require handling.[35][32] |
| Apple Vision Pro | System interaction combines gaze targeting and finger gestures; ARKit offers a separate skeletal-data provider. | Ordinary system selection and authorized immersive hand-joint access are different input paths.[21][20] |
| Ultraleap | Dedicated stereo infrared hand cameras with a software hand model. | Hyperion packages include a tracking service, LeapC SDK and control panel; model, hint and performance-mode availability depends on configuration, with some features marked beta.[36] |
| Varjo integrations | Ultraleap documentation lists built-in tracking on XR-3, VR-3 and VR-2 Pro. | XR-4 uses an optional hand-tracking module; the integration is model-specific.[37] |
| MediaPipe Hand Landmarker | Image-based hand detection and landmark estimation. | A vision component, requiring additional XR-space registration and interaction logic for a complete immersive input system.[24][27] |
The hand displayed while holding a controller can have a different data source from an optically tracked bare hand. Meta's controller-driven hand-pose documentation uses capacitive contact and button states to infer finger configurations. It offers Natural Hand Poses, which can show hands while hiding controllers, and Controller Hand Poses for a controller representation. These configurations are useful for animation and input, but cannot establish the exact physical angle of every finger joint.[38]
Multimodal tracking also has feature conflicts. Meta's dedicated documentation identifies incompatibilities with selected body-tracking and Fast Motion Mode configurations, and describes limitations in controller-in-hand detection and Meta Link behavior. A visually plausible hand/controller transition can still be incorrect. The generic claim that hands and controllers can never be tracked concurrently is therefore too broad, while enabling one feature does not guarantee every combination is supported.[34]
Permissions and data access
The availability of a hand to the operating system does not necessarily grant an application access to its skeleton. Apple's original ARKit presentation distinguishes system gestures from HandTrackingProvider data, requires hand-tracking authorization for that provider and describes handling denial.[20] WebXR's June 2024 draft restricts hand data to immersive sessions with the requested feature and explicit consent; it also identifies privacy implications of revealing a user's hand characteristics. These are specification requirements, rather than proof of identical consent interfaces across all browsers.[31]
Meta's September 2026 Unity overview restricts the use of hand-tracking data to hand-tracking functionality. It also describes Wide Motion Mode as extending plausible poses beyond the camera view. A skeleton returned under such a mode remains an estimate and should not be silently reused as biometric or research ground truth.[33] Applications need to distinguish provider availability, authorization, tracked status and the action the user intended to perform. These are separate conditions: a supported provider can be unauthorized, and an authorized provider can temporarily lack a tracked hand.[20][28]
Interaction paradigms
Direct manipulation
Direct manipulation associates a visible hand or fingertip with a nearby object. HoloLens 2 design guidance describes pressing virtual buttons, touching slates and grabbing manipulation handles. Visual proximity effects, button depression and audio communicate approaching, touching and pressing; a bare hand does not supply the physical resistance of a real switch. The same guidance describes two-hand movement, rotation and scaling of an object's bounds. These are interaction techniques layered on tracking data.[35]
The distinction between sensing and task behavior is visible in controlled experiments. Masurovsky and colleagues compared basic Leap Motion grabbing, a modified hand interaction and Oculus Touch in a 2020 grab-and-place study with 32 participants. Controllers performed better on the tested task measures, while modifications to the hand interaction reduced unintended drops relative to the basic version. Changes included constraints on when a grab could begin and additional feedback. Their combined design did not demonstrate that every individual modification independently improved performance, or that the result applies to all modern hand tracking.[39]
Ray-based selection (indirect interaction)
An indirect technique separates the hand's location from the selected object. A hand ray can point to a distant interface, while a pinch confirms the selection. Gaze-and-confirm schemes separate targeting still further. Meta's design guidance describes direct touch/grab, rays, gaze-assisted selection and microgestures as choices for different task layouts. The guidance recommends a small interaction set and a priority rule when techniques overlap.[40]
Microsoft's interactable-object guidance distinguishes direct hand targets from ray or gaze targets. It recommends approximately 1.6 x 1.6 cm direct targets at 45 cm and 3.5 x 3.5 cm targets at 2 m for ray/gaze examples, alongside angular sizing guidance. These are platform design recommendations with explicit distances, not universal minimums guaranteed to work with every sensor or user. It also describes hover, press and other state changes so users can see when input has been interpreted.[41]
Multimodal interaction
Hand tracking can combine with eye tracking, voice, controllers or physical objects, but those combinations need an implemented interaction model. On Vision Pro, Apple describes looking at an item and tapping thumb and index finger to select it, with pinch-and-drag for movement and a flick for scrolling. Hands need not be held directly in front of the face.[21] On Quest, dedicated Multimodal support allows hands and controllers within listed configuration boundaries.[34] These examples do not establish one universal set of gestures across platforms.
A controller remains useful when a task needs physical buttons or a held grip. Capacitive contact can animate a hand around that controller, while a camera-tracked hand can support an empty-hand action. Treating both rendered skeletons as equivalent anatomical observations would erase the distinction between sensed contacts and estimated free-hand articulation.[38]
Gesture commands
A gesture vocabulary should state what users do and what action follows. Pinching, pointing or grabbing can mean different things in different contexts. Meta's interaction-model guidance stresses matching the technique to the layout and reducing ambiguity when modes overlap.[40] Recognition can also fail because the pose is not available: limited camera visibility, crossing hands or covering them with an object affects the underlying estimate before the command layer interprets it.[42]
Defined hand alphabets are a narrower historical example. Grimes's glove patent mapped sensed configurations to data-entry characters.[6] That mechanism should not be described as a complete translator simply because it used a manual alphabet. Hand pose input, command recognition and the interpretation of a language are different application tasks.
Meta's v50 Direct Touch feature used finger contact for buttons, scrolling and virtual-keyboard interaction.[19] Vision Pro support guidance describes both eye-and-pinch selection and touching virtual keyboard keys with one finger from each hand. Gloves, long sleeves, large jewelry and crossed hands can interfere with the device's ability to see the intended gestures. These are specific troubleshooting conditions rather than a single visibility rule for every optical tracker.[21]
Hand-attached menus offer quickly accessible actions, but sustained use can require an uncomfortable arm position. Microsoft recommends keeping such menus small, using them for brief actions and providing alternatives such as voice or a menu that can be placed in the world. Its guidance also warns against assuming every user can operate both hands. A menu's reach and placement are therefore part of input accessibility, even when the skeleton is tracked accurately.[43]
Applications
Gaming and creative interaction
Hand Physics Lab is an example of applying tracked hands to a physics-based environment. Its developer lists interactions with buttons, tools and building blocks, drawing with a pen and painting with the fingers. The product FAQ checked in October 2026 lists both hand and controller input on supported Quest devices and hand input on Apple Vision Pro. The application's physics and gesture interpretation operate above the device's hand estimate.[44]
Object interaction, creation and training interfaces
HoloLens 2 design guidance describes model manipulation through bounds handles, direct hands and hand rays.[35] An evaluation should distinguish whether the system recognized a pose from whether the user completed the intended operation. The 2020 grab-and-place experiment, for example, measured unintended drops and task performance rather than merely whether a hand appeared on screen.[39]
Hand input also changes how an application communicates. Buckingham's 2021 perspective discusses nonverbal expression, showing real-world gesture habits in VR and the potential contribution to embodiment. It also raises missing touch feedback and mismatches between an avatar hand and the user's movement. These are design opportunities and challenges; they do not prove that every application creates stronger presence or better learning than a controller interface.[3]
Measurement and rehabilitation research
Research can use hand tracking to assess movement, but clinical claims require a validated task and population. Everard and colleagues' 2024 cross-sectional study compared the classic Box and Block Test with controller-based VR, hand-tracked VR and mixed-reality versions. It included 21 participants with stroke, 21 controls and usability input from five therapists. Participants had to be able to move at least one block and understand directions. The Quest 2 hand-tracked test had encouraging concurrent validity and same-session reliability in that protocol. This supports study of a dexterity assessment, not a claim that hand tracking treats stroke or that a virtual test is interchangeable with every clinical measure.[45]
Measurement also depends on processing choices. The 2025 HoloLens 2 study used filtering and offline alignment before comparison with Vicon, and its pinch protocol compared measurements from different hands. These choices matter when assessing whether an application requires joint-angle measurement, live feedback or object interaction.[46] Device-level tracking precision alone cannot establish a rehabilitation intervention's effectiveness.
Human-computer interaction and accessibility
Hands can provide input without a separate held controller, but accessible placement and alternate input still matter. Ultraleap's wheelchair playtesting account recommends adjustable height and reach, interaction that does not assume standing, and consideration of hands needed to move a wheelchair. It is a practical account of accessibility barriers, not a trial proving universal access.[47] Microsoft's hand-menu guidance similarly calls for one-hand and voice alternatives.[43]
Hands also provide a subject for human-computer interaction research. Schneider and colleagues' 2021 touch-task study compared several historical headset/hand-tracking integrations using 20 participants, then examined a fixed-head setup with a robot and an artificial hand. The different tasks and calibration procedures affected the interpretation of measured errors. Such studies help investigate a particular interface, but are not rankings that can be carried forward unchanged after hardware or software updates.[48]
Motion recording and embodied input
Ingram's calibrated glove recordings examined hand-movement covariation during daily activities.[9] Optical skeletons can also drive visual hands, while controller contact states can supply an inferred pose for an avatar holding a device.[38] Applications should retain that data-source distinction when storing motion, comparing participants or interpreting an avatar's movements.
Performance and accuracy
A hand tracker has several measurable properties. Position error, joint-angle error, repeatability, tracking availability, gesture success and latency describe different outcomes. A result obtained after fitting the tracked coordinates to a reference is also different from an absolute error in a previously calibrated coordinate frame. Published experiments therefore need their reference system, processing, software version and task attached to their numbers.[49][46]
| Original study | Tested configuration and reference | Reported result | Interpretation boundary |
|---|---|---|---|
| Weichert et al. (2013), Leap Motion | Preliminary developer kit; robot-guided reference pen; controlled lighting and temperature. | Static deviation below 0.2 mm; dynamic mean approximately 1.2 mm. | Pen tracking, not a live anatomical hand-joint guarantee; authors could not substantiate 0.01 mm accuracy.[11] |
| Abdlkarim et al. (online 2023, issue 2024), Meta Quest 2 | Eight adults; build 33.0; headset rigidly mounted above the hand workspace; Qualisys reference. | Mean static position error 1.1 cm, mean angular error 9.6 degrees and estimated absolute delay 45 ms. | Historical fixed-headset laboratory protocol; spatial error varied across the workspace.[49] |
| Bertolasi et al. (2025), HoloLens 2 | Ten participants in tracing/grasp experiments; Vicon reference; stationary head; low-pass filtering, temporal alignment and iterative closest-point registration. | Mean fingertip tracing error about 2.8 mm for most fingers and 3.9 mm for the index finger after that processing. | Aligned, filtered trajectories rather than an unconditional live spatial-accuracy or latency specification.[46] |
The Quest 2 framework tested multiple target heights and locations, selected joint angles and reaching movements. Spatial error increased toward the workspace periphery. Rigid mounting limits extrapolation to normal head movement. The estimated 45 ms delay included a 7 ms reference-system allowance.[49]
The HoloLens 2 comparison was published on July 3, 2025. Both streams were filtered, time-aligned and registered before fingertip trajectory errors were calculated. Its angular errors varied across joints and grasps; its small pinch experiment used different hands for the two measurement systems. Consequently, low fitted tracing error should not be converted into an anatomical angle guarantee or a clinical claim.[46]
Schneider's 2021 touch study provides another distinction. It used the original Quest with OS 25, HoloLens 2 with Windows 19041.1136, Magic Leap 1 with LuminOS 0.98.20 and contemporary VIVE/Leap integrations. Participants could use a visible virtual fingertip to compensate while touching targets; the robot experiment placed an artificial hand at fixed physical positions without that compensation. The authors discuss these differences and other confounds. A human's successful target acquisition can therefore include adaptation to a displayed offset, whereas a robot comparison measures a different behavior.[48]
Timestamps, prediction and calibration
An application may receive a pose projected toward display time, which is useful for interaction but differs from a capture-time estimate. Meta's October 7, 2026 unextrapolated-pose documentation distinguishes predicted GetHandState output from GetUnextrapolatedHandState. The latter requires OVRPlugin 1.116.0 or later, an OpenXR backend and OpenXR hand skeleton in the documented configuration. It supplies a tracking estimate without that prediction, rather than raw camera images or guaranteed anatomical truth. Mixing predicted hand samples with differently timed reference data can change the apparent error.[50]
Spatial registration also needs attention. A sensor's mounting offset, the XR reference space and the measurement reference are separate parts of the setup. Ultraleap's integration guidance recommends using its Visualizer to inspect tracking and setting the intended desktop/head-mounted configuration; it also discusses physical offsets between the sensor and eyes.[26] OpenXR explicitly locates joints to a base space at a requested time. A reproducible comparison must identify both choices.[27]
Glove calibration and offline camera-trajectory registration serve different purposes. Their procedures should accompany reused data.[9][46]
Advantages
Direct-manipulation interfaces support familiar actions with visual and audio feedback.[35] Distant content can instead use rays or gaze-assisted confirmation, reducing the requirement to reach each target physically. The choice depends on the layout and the user's intended task.[40]
Visible hands can convey nonverbal expression and provide an embodiment cue, as discussed in Buckingham's perspective.[3] Gloves can record articulation during daily activities when fitted and calibrated.[9] These benefits do not establish a universal speed, precision or comfort advantage over controllers. In the controlled 2020 grab-and-place task, physical controllers still produced better results than the tested hand interactions.[39]
Challenges and limitations
Technical limitations
Occlusion and viewing volume. A hand can cover another finger, another hand or an object, leaving the tracker with incomplete evidence. Meta's limitations guide describes confidence changes and fading during occlusion, and distinguishes the viewing conditions for hands from those for head tracking.[42] MEgATrack's original paper reported severe hand-object failures despite model constraints.[15] Invisible joints may be inferred rather than measured; availability alone does not show that all parts were seen.
Valid versus tracked data. OpenXR distinguishes valid output from tracked output, and can emulate an unobstructed range when the available input is constrained.[28] A historical HoloLens 2 December 2020 release note described retaining palm tracking while inferring other joints from the previous pose when a hand was occluded, with High and Approximate accuracy classifications. That release-specific description is an example of how an apparently complete hand can contain estimates; it should not be imposed on every later API.[51]
Environment and hardware integration. Lighting, hand visibility and objects covering the hands can affect optical tracking.[42] Vision Pro visibility limits are described above. Ultraleap's integration guidance identifies USB bandwidth and dirty camera covers as additional issues. A tracking problem can therefore originate in installation or data transfer as well as in the pose model.[26]
Recovery and application state. An input provider may lose a hand or input focus. OpenXR defines an inactive state with cleared validity flags, while archived MRTK2 documentation warns about failed joint queries and destroyed joint objects. Applications need to respond to those states rather than applying a stale pose to a continuing command.[28][32] MediaPipe's detector/tracker separation similarly uses a recovery path when preceding landmarks no longer provide a reliable crop.[24]
Representation mismatch. A model's joint count and coordinate convention may differ from another API. OpenXR's 26-entry skeleton and WebXR's 25-entry skeleton include different palm treatment.[25][31] Controller contact-based finger animation is another distinct data source.[38] Combining these inputs requires explicit mapping and data-source identification; a consistent-looking avatar is insufficient evidence of consistent measurement.
User experience challenges
Direct-manipulation guidance uses visual and audio contact cues; hand-menu guidance encourages brief use and alternate input.[35][43] The absence of a held controller does not remove a user's reach, fatigue or motor-access requirements.
Gesture ambiguity also depends on context. A pinch can be a command, a grab or an incidental movement, and simultaneous direct and indirect modes can target different objects. Meta's guidance therefore emphasizes choosing a coherent small vocabulary and resolving competition between input models.[40] The grab-and-place study's unintended-drop results show why task-level failures need evaluation alongside pose tracking.[39]
Accessibility cannot be inferred from the phrase "natural input". A wheelchair user may need a reachable menu and free hands for mobility; a user who operates one hand needs an alternative to a required two-hand gesture.[47][43] Buckingham's perspective raises broader inclusion questions, but it does not measure a device-specific demographic disparity.[3] Testing an application's actual gestures, positions and fallback controls with its intended users remains distinct from measuring a skeletal model's average error.
Future developments
Neural interfaces
Electromyography (EMG) measures muscle electrical activity and offers a different input source from cameras. In a 2025 Nature paper, Kaifosh, Reardon and CTRL-labs described a non-invasive surface-EMG research interface with 16 bipolar channels sampled at 2 kHz. Population-trained models supported wrist control, discrete gestures and handwriting in online experiments with new users. The input was peripheral muscle activity, not direct reading of thoughts or every anatomical finger position. Its research hardware and tasks do not establish specifications for every commercial wristband.[52]
Muscle input and optical hand input could serve different interaction needs, but evaluating one does not automatically validate the other. The Nature research reports task throughput and investigates generalization across participants. Those outcomes are different from a claim of zero or negative end-to-end interaction latency.[52]
Advanced haptics
Tracking identifies an input pose; haptic feedback supplies a sensation. Existing approaches already combine these functions. HaptX's G1 specification lists 135 tactile actuators and five force actuators per glove alongside its motion-capture subsystem, airpack and tethers. Its footnotes distinguish mobile best-case feedback timing from desktop configurations with longer tubes. Those figures describe a particular 2024 product sheet, rather than a universal glove response.[29]
Ultraleap's mid-air haptics documentation describes phased ultrasound arrays concentrating waves at modulated control points. A hand-tracking camera positions those points on the palm or fingers; amplitude and spatiotemporal modulation produce localized sensations. This adds cutaneous feedback without a worn glove, but is not a physical surface that constrains a grasp like a rigid object.[53] Designing contact feedback therefore involves both where the hand is estimated to be and what the feedback hardware can produce.
Neural networks and prediction
Current systems combine learned image models with temporal information, but prediction does not remove every ambiguity. MediaPipe's original system reused preceding landmarks and redetected a lost hand.[23] MEgATrack used history and geometric constraints while still failing in difficult occlusions.[15] Ultraleap Hyperion documentation exposes model, hint and performance-mode choices, including beta features whose scope differs from ordinary production tracking.[36]
A useful evaluation distinguishes newly visible evidence, inferred hidden joints, predicted future pose and application action success. Meta's separate unextrapolated-pose API makes one of these timing distinctions explicit.[50] Reproducible measurements and documented configuration changes make the behavior of successive implementations easier to compare.
See also
References
- ↑ 1.0 1.1 1.2 "Ultraleap Hand Tracking Overview". Ultraleap documentation. https://docs.ultraleap.com/hand-tracking/index.html. Retrieved 2026-10-12.
- ↑ 2.0 2.1 2.2 "CyberGlove II". CyberGlove Systems. https://www.cyberglovesystems.com/cyberglove-ii/. Retrieved 2026-10-12.
- ↑ 3.0 3.1 3.2 3.3 "Hand Tracking for Immersive Virtual Reality: Opportunities and Challenges". Frontiers in Virtual Reality. 2021-10-20. https://www.frontiersin.org/journals/virtual-reality/articles/10.3389/frvir.2021.728461/full. Retrieved 2026-10-12.
- ↑ 4.0 4.1 "Final Report to the National Endowment of the Arts (US NEA R60-34-163)". Thomas A. DeFanti and Daniel J. Sandin, University of Illinois at Chicago Circle. 1977-11. https://www.evl.uic.edu/documents/us_nea_r60-34-163_sayre_glove_finalreport_november1977.pdf. Retrieved 2026-10-12.
- ↑ 5.0 5.1 5.2 "17.4 Interaction". Computer Graphics and Computer Animation: A Retrospective Overview. https://ohiostate.pressbooks.pub/graphicshistory/chapter/1-4-interaction/. Retrieved 2026-10-12.
- ↑ 6.0 6.1 6.2 "US4414537A - Digital data entry glove interface device". Google Patents. 1983-11-08. https://patents.google.com/patent/US4414537A/en. Retrieved 2026-10-12.
- ↑ 7.0 7.1 "US4542291A - Optical flex sensor". Google Patents. 1985-09-17. https://patents.google.com/patent/US4542291A/en. Retrieved 2026-10-12.
- ↑ 8.0 8.1 8.2 "'PowerGlove' computer game accessory". Powerhouse Collection. 1989. https://collection.powerhouse.com.au/object/166845. Retrieved 2026-10-12.
- ↑ 9.0 9.1 9.2 9.3 9.4 9.5 "The statistics of natural hand movements". Experimental Brain Research. 2008. https://pmc.ncbi.nlm.nih.gov/articles/PMC2636901/. Retrieved 2026-10-12.
- ↑ 10.0 10.1 "5DT Data Glove Ultra". 5DT. https://5dt.com/5dt-data-glove-ultra/. Retrieved 2026-10-12.
- ↑ 11.0 11.1 11.2 "Analysis of the Accuracy and Robustness of the Leap Motion Controller". Sensors. 2013-05-14. https://pmc.ncbi.nlm.nih.gov/articles/PMC3690061/. Retrieved 2026-10-12.
- ↑ 12.0 12.1 "Realtime and Robust Hand Tracking from Depth". Proceedings of CVPR 2014. 2014. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/yichenw-cvpr14_handtracking.pdf. Retrieved 2026-10-12.
- ↑ 13.0 13.1 13.2 13.3 "Hand Tracking SDK for Oculus Quest Available with v12 Release". Meta for Developers. 2019-12-09. https://developers.meta.com/vr/blog/hand-tracking-sdk-for-oculus-quest-available/. Retrieved 2026-10-12.
- ↑ "Useful Links". Ultraleap documentation. https://docs.ultraleap.com/useful-links.html. Retrieved 2026-10-12.
- ↑ 15.0 15.1 15.2 15.3 15.4 "MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality". ACM Transactions on Graphics 39(4), article 87. 2020. https://dl.acm.org/doi/10.1145/3386569.3392452. Retrieved 2026-10-12.
- ↑ "Upgrading to Ultraleap Gemini V5.2+". Ultraleap documentation. https://docs.ultraleap.com/hand-tracking/gemini-migration.html. Retrieved 2026-10-12.
- ↑ 17.0 17.1 "Dive into the Latest Hand Tracking Improvements with v2.1 Update". Meta for Developers. 2023-01-19. https://developers.meta.com/horizon/blog/hand-tracking-improvements-v2-1/. Retrieved 2026-10-12.
- ↑ "All Hands on Deck: Crank up Hand Responsiveness and Unlock New Gameplay with Hands 2.2". Meta for Developers. 2023-08-21. https://developers.meta.com/vr/blog/hand-tracking-22-response-time-meta-quest-developers/. Retrieved 2026-10-12.
- ↑ 19.0 19.1 19.2 "Use Your Fingers (Not Controllers) to Swipe Through the VR Interface on Meta Quest". Meta Newsroom. https://about.fb.com/news/2023/02/meta-quest-direct-touch-use-your-fingers-in-vr/. Retrieved 2026-10-12.
- ↑ 20.0 20.1 20.2 20.3 20.4 20.5 "Meet ARKit for spatial computing". Apple Developer, WWDC23. 2023. https://developer.apple.com/videos/play/wwdc2023/10082/. Retrieved 2026-10-12.
- ↑ 21.0 21.1 21.2 21.3 "Use gestures with Apple Vision Pro". Apple Support. 2026-09-28. https://support.apple.com/en-us/117741. Retrieved 2026-10-12.
- ↑ 22.0 22.1 "Khronos Releases OpenXR 1.1 to Further Streamline Cross-Platform XR Development". Khronos Group. 2024-04-15. https://www.khronos.org/news/press/khronos-releases-openxr-1.1-to-further-streamline-cross-platform-xr-development. Retrieved 2026-10-12.
- ↑ 23.0 23.1 23.2 "MediaPipe Hands: On-device Real-time Hand Tracking". Google Research, arXiv:2006.10214. 2020-06-18. https://arxiv.org/pdf/2006.10214. Retrieved 2026-10-12.
- ↑ 24.0 24.1 24.2 "Hand landmarks detection guide". Google AI Edge. 2026-10-01. https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker. Retrieved 2026-10-12.
- ↑ 25.0 25.1 "XrHandJointEXT(3) Manual Page". Khronos Group. https://registry.khronos.org/OpenXR/specs/1.1/man/html/XrHandJointEXT.html. Retrieved 2026-10-12.
- ↑ 26.0 26.1 26.2 "Integration tips and resources". Ultraleap documentation. https://docs.ultraleap.com/hand-tracking/integration-tips-resources.html. Retrieved 2026-10-12.
- ↑ 27.0 27.1 27.2 "xrLocateHandJointsEXT(3) Manual Page". Khronos Group. https://registry.khronos.org/OpenXR/specs/1.1/man/html/xrLocateHandJointsEXT.html. Retrieved 2026-10-12.
- ↑ 28.0 28.1 28.2 28.3 28.4 "XrHandJointLocationsEXT(3) Manual Page". Khronos Group. https://registry.khronos.org/OpenXR/specs/1.1/man/html/XrHandJointLocationsEXT.html. Retrieved 2026-10-12.
- ↑ 29.0 29.1 "HaptX Gloves G1 Specifications". HaptX, revision 1.4. 2024. https://docs.haptx.com/docs/files/HaptX_Glove_G1_Spec_Sheet_Rev_1.4_Spec.pdf. Retrieved 2026-10-12.
- ↑ "OpenXR Specification: XR_EXT_hand_tracking". Khronos Group. https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XR_EXT_hand_tracking. Retrieved 2026-10-12.
- ↑ 31.0 31.1 31.2 "WebXR Hand Input Module - Level 1". World Wide Web Consortium. 2024-06-05. https://www.w3.org/TR/webxr-hand-input-1/. Retrieved 2026-10-12.
- ↑ 32.0 32.1 32.2 "Hand tracking — MRTK2". Microsoft Learn. 2022-08-02. https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/features/input/hand-tracking?view=mrtkunity-2021-05. Retrieved 2026-10-12.
- ↑ 33.0 33.1 "Hand Tracking Overview for Meta VR Devices in Unity". Meta Horizon Developer documentation. https://developers.meta.com/vr/documentation/unity/unity-handtracking-overview/. Retrieved 2026-10-12.
- ↑ 34.0 34.1 34.2 "Use Simultaneous Hands and Controllers (Multimodal)". Meta Horizon Developer documentation. 2026-10-07. https://developers.meta.com/vr/documentation/unity/unity-multimodal/. Retrieved 2026-10-12.
- ↑ 35.0 35.1 35.2 35.3 35.4 "Direct manipulation with hands". Microsoft Learn. https://learn.microsoft.com/en-us/windows/mixed-reality/design/direct-manipulation. Retrieved 2026-10-12.
- ↑ 36.0 36.1 "Ultraleap Hyperion Overview". Ultraleap documentation. https://docs.ultraleap.com/hand-tracking/Hyperion/index.html. Retrieved 2026-10-12.
- ↑ "Ultraleap for Varjo". Ultraleap documentation. https://docs.ultraleap.com/varjo/index.html. Retrieved 2026-10-12.
- ↑ 38.0 38.1 38.2 38.3 "Use Capsense". Meta Horizon Developer documentation. 2026-08-11. https://developers.meta.com/vr/documentation/unity/unity-capsense/. Retrieved 2026-10-12.
- ↑ 39.0 39.1 39.2 39.3 "Controller-Free Hand Tracking for Grab-and-Place Tasks in Immersive Virtual Reality: Design Elements and Their Empirical Study". Multimodal Technologies and Interaction 4(4), 91. 2020-12-12. https://pure.mpg.de/rest/items/item_3293163_2/component/file_3293164/content. Retrieved 2026-10-12.
- ↑ 40.0 40.1 40.2 40.3 "Choosing an interaction model". Meta Horizon Developer documentation. 2026-09-29. https://developers.meta.com/vr/design/hands-choosing-an-interaction-model/. Retrieved 2026-10-12.
- ↑ "Interactable object". Microsoft Learn. https://learn.microsoft.com/en-us/windows/mixed-reality/design/interactable-object. Retrieved 2026-10-12.
- ↑ 42.0 42.1 42.2 "Troubleshooting and Limitations". Meta Horizon Developer documentation. 2026-09-14. https://developers.meta.com/vr/documentation/unity/unity-handtracking-troubleshooting-limitations/. Retrieved 2026-10-12.
- ↑ 43.0 43.1 43.2 43.3 "Hand menu". Microsoft Learn. https://learn.microsoft.com/en-us/windows/mixed-reality/design/hand-menu. Retrieved 2026-10-12.
- ↑ "Hand Physics Lab". Holonautic. https://www.holonautic.com/hand-physics-lab. Retrieved 2026-10-12.
- ↑ "Extended reality to assess post-stroke manual dexterity: contrasts between the classic box and block test, immersive virtual reality with controllers, with hand-tracking, and mixed-reality tests". Journal of NeuroEngineering and Rehabilitation 21, 36. 2024-03-15. https://pmc.ncbi.nlm.nih.gov/articles/PMC10941416/. Retrieved 2026-10-12.
- ↑ 46.0 46.1 46.2 46.3 46.4 "Evaluation of HoloLens 2 for Hand Tracking and Kinematic Features Assessment". Virtual Worlds 4(3), 31. 2025-07-03. https://www.mdpi.com/2813-2084/4/3/31. Retrieved 2026-10-12.
- ↑ 47.0 47.1 "VR Accessibility Recommendations". Ultraleap documentation. https://docs.ultraleap.com/ultralab/vr-gaming-accessibility-recommendations.html. Retrieved 2026-10-12.
- ↑ 48.0 48.1 "Accuracy Evaluation of Touch Tasks in Commodity Virtual and Augmented Reality Head-Mounted Displays". ACM Symposium on Spatial User Interaction 2021, author preprint. 2021. https://arxiv.org/pdf/2109.10607. Retrieved 2026-10-12.
- ↑ 49.0 49.1 49.2 "A methodological framework to assess the accuracy of virtual reality hand-tracking systems: A case study with the Meta Quest 2". Behavior Research Methods 56, 1052-1063. 2023-02-13. https://pmc.ncbi.nlm.nih.gov/articles/PMC10830632/. Retrieved 2026-10-12.
- ↑ 50.0 50.1 "Use Unextrapolated Hand Poses". Meta Horizon Developer documentation. 2026-10-07. https://developers.meta.com/vr/documentation/unity/unity-handtracking-unextrapolated-poses/. Retrieved 2026-10-12.
- ↑ "HoloLens 2 release notes - Windows Holographic, version 2004". Microsoft Learn. 2020. https://learn.microsoft.com/en-us/hololens/hololens-release-notes-2004. Retrieved 2026-10-12.
- ↑ 52.0 52.1 "A generic non-invasive neuromotor interface for human-computer interaction". Nature 645, 702-711. 2025-07-23. https://www.nature.com/articles/s41586-025-09255-w. Retrieved 2026-10-12.
- ↑ "Ultraleap mid-air haptics". Ultraleap documentation. https://docs.ultraleap.com/haptics/index.html. Retrieved 2026-10-12.