Face tracking

See also: Terms and Technical Terms

Face tracking (also written face-tracking or facial tracking, and sometimes called facial expression tracking) is a sensor technology that estimates the movements and expressions of a user's face, such as the mouth, lips, jaw, cheeks, and brows, and converts them into data that can drive an avatar or interface in real time. In the context of virtual reality (VR) and augmented reality (AR), it refers to the integration of inward-facing cameras or depth sensors within a head-mounted display (HMD), or in an add-on accessory, to reproduce a wearer's facial movements on a digital representation of themselves. The goal is to raise social presence and the sense of non-verbal communication in shared virtual spaces by letting a person smile, frown, talk, or raise an eyebrow and have their avatar do the same.

Face tracking is distinct from two related sensing modalities that are often discussed alongside it. Eye tracking measures where the user is looking (gaze) using the pupil and cornea, whereas face tracking is concerned with the muscles and surfaces of the rest of the face. Hand tracking estimates the pose of the hands and fingers. Some high-end headsets combine all three, treating eye tracking, face tracking, and hand tracking as separate inputs that together produce a more lifelike avatar and more natural interaction. Because facial movement is closely tied to identity, emotion, and speech, face tracking also raises privacy questions about how facial data is captured, processed, and stored.

Background

The representation underlying most modern face tracking traces to the Facial Action Coding System (FACS), a taxonomy of human facial movement. The psychologists Paul Ekman and Wallace V. Friesen first published the FACS manual in 1978. They built on earlier work by the anatomist Carl-Herman Hjortsjö, author of Man's Face and Mimic Language, who in the Paul Ekman Group's words "did important groundwork in identifying the units of action". Ekman, Friesen and Joseph C. Hager published a significant update in 2002, which the Paul Ekman Group describes as the current version of the manual. FACS describes facial movement in terms of "action units" (AUs), the fundamental actions of individual muscles or groups of muscles. Each action unit is identified by a number and a short name, and its anatomical basis is given as the contraction or relaxation of one or more facial muscles, for example the inner brow raiser (AU 1), the nose wrinkler (AU 9) and the jaw drop (AU 26). The system was designed to distinguish all visible, anatomically based facial movements, and it contains no emotion-specific descriptors, although it is commonly used to interpret facial expressions related to emotion and other human states.

In computer animation, the Facial Action Coding System has been used to guide the construction of blendshape models, in which each target shape (also called a morph target) is a facial expression or a localized movement such as raising one eyebrow. A face is produced as a linear combination of these targets: each target carries a weight, conventionally limited to the range 0 to 1, and the weighted differences between the targets and a neutral face are added together. When two face models have targets with the same meaning, typically inspired by FACS, the weights can simply be copied from one model to the other. VR face tracking systems use the same representation: Meta's Movement SDK categorizes facial movements into expressions based on FACS and reports each as a blendshape with a strength that indicates its activation, which lets the same tracking data drive many different avatar designs.

How it works

Face tracking in an HMD generally relies on one of two approaches: optical sensing of the face from cameras placed inside or beneath the headset, or, where no such cameras exist, inference of facial movement from other signals such as the microphone.

Optical (camera-based) tracking

Camera-based systems place small infrared (IR) cameras and IR illuminators so they can see parts of the face that the headset normally hides. Cameras pointed at the eyes and brows capture upper-face movement, while a downward or forward camera (often in an add-on under the visor) captures the lower face: the lips, cheeks, jaw, and sometimes the tongue. Computer vision algorithms analyze these images and output a stream of blendshape weights describing the current expression. IR illumination is used because it is invisible to the wearer and gives consistent imaging inside the dark interior of a headset. Depth-sensing front cameras, such as those used to scan the face before a session, can additionally build a 3D model of the user's features for a more lifelike result.

Audio-based estimation

Headsets that lack inward-facing cameras can still animate a mouth by estimating lip and jaw movement from the user's speech captured by the microphone. This audio-driven approach, sometimes called lip sync, does not see the face and cannot reproduce expressions that make no sound (for example a silent smile or a raised eyebrow), so it is less expressive than camera-based tracking. Meta uses microphone-based estimation for Meta Quest 2 and Meta Quest 3, which have no inward-facing cameras, and reserves true camera-based face tracking for headsets that do.

Representation and data levels

Whichever sensing method is used, the output is normally a set of named blendshapes with activation strengths rather than raw imagery. Meta's Movement SDK, for example, exposes facial expressions through OpenXR as defined blendshapes whose strength indicates activation; its expression sets include on the order of 63 to 71 named blendshapes covering the brows, cheeks, jaw, lips, and (in the larger set) the tongue. Apple's ARKit, used on iPhone and iPad for face-based AR, reports 52 blendshape coefficients (each 0.0 to 1.0) derived from the TrueDepth camera at 60 frames per second, which developers map onto an avatar or "puppet" that follows the user's expressions.

Devices and implementations

Face tracking is far less common in consumer headsets than eye tracking, and several popular VR products that include eye tracking do not track facial expression at all.

Headsets with built-in face tracking

• Meta Quest Pro: Released in October 2022, the Quest Pro was, in Meta's words, "the first headset we've built that integrates inward-facing sensors to capture natural facial expressions and eye tracking." Its face-tracking feature, branded Natural Facial Expressions, uses inward-facing cameras to detect facial movement, which the system converts into activations of FACS-based blendshapes such as jaw drop and nose wrinkle so that an avatar can mirror the wearer. The feature is off by default and the user chooses whether to enable it. Meta states that the images the headset captures of the face never leave the device and are deleted after processing, so neither Meta nor third-party apps receive the raw images.
• Apple Vision Pro: Apple's headset does not animate a generic cartoon avatar but instead drives a Persona, a photorealistic digital likeness of the wearer used in FaceTime and other video apps. The Persona is created from a one-time facial scan in which the device's cameras "capture images and 3D measurements of your face, head, upper body, and facial expressions," and it is then animated live during a call to reflect the user's expressions and head movement. Apple states that the data used to build the Persona does not leave the device, and that during a call the user's Persona is sent securely to the other participants.

Add-on facial trackers

For headsets without integrated face tracking, manufacturers have sold accessories that clip onto the underside of the visor to capture the lower face.

• HTC VIVE Facial Tracker: An add-on for the VIVE Pro and compatible PC VR headsets that tracks up to 38 facial movements across the lips, jaw, teeth, tongue, chin, and cheeks. It uses dual cameras with IR illumination at a 60 Hz tracking rate and a sub-10-millisecond response time, connecting over a built-in USB-C cable, with data exposed through HTC's SRanipal SDK for Unity and Unreal Engine. HTC suggested pairing it with the eye-tracking VIVE Pro Eye for full-face capture. This original tracker has since been discontinued.
• VIVE Focus / XR Elite facial trackers: HTC later introduced facial trackers for its standalone line. The Facial Tracker for the VIVE Focus series captures expressions through 38 blendshapes across the lips, jaw, cheeks, chin, teeth, and tongue at 60 Hz, and the VIVE Full Face Tracker for the VIVE XR Elite, shown at CES 2024, combines lower-face tracking with eye tracking and automatic interpupillary distance calibration.

Headsets without face tracking

A number of devices include eye tracking but no facial expression tracking. The PlayStation VR2 is one example. Sony's specifications list four embedded cameras for headset and controller tracking and an IR camera for eye tracking for each eye; no camera for the lower face is listed. The eye-tracking data can be used for foveated rendering, in which only the area the user is looking at is rendered in full detail. Sony also describes eye tracking as an extra input, so that "a simple look in a specific direction can create an additional input for the game character", and says that in compatible games it can make the player's avatar show heightened emotional responses. The eye-tracking technology comes from Tobii, which announced in February 2022 that it was negotiating with Sony Interactive Entertainment to supply it and confirmed the deal in a press release reported by UploadVR on 30 June 2022. Likewise, the Meta Quest 2 and Meta Quest 3 have no inward-facing cameras; Meta estimates facial movement on them from microphone audio instead.

The table below summarizes facial sensing on several current devices.

Device	Eye tracking	Face (expression) tracking	Method
Meta Quest Pro	Yes	Yes (Natural Facial Expressions)	Inward-facing cameras, FACS blendshapes
Apple Vision Pro	Yes	Yes (drives a Persona)	Cameras plus depth sensing, on-device
PlayStation VR2	Yes (Tobii)	No	One IR eye-tracking camera per eye, used for foveated rendering
Meta Quest 2 / Meta Quest 3	No	No (audio lip estimation only)	Microphone-based mouth estimation
HTC VIVE Pro + VIVE Facial Tracker	Via VIVE Pro Eye	Yes (lower face, add-on)	Dual IR cameras, 38 movements, 60 Hz

Applications

Avatar expression in social VR

The primary use of face tracking is to animate a user's avatar in social VR so that conversation carries facial cues, not just voice. Meta promotes Natural Facial Expressions for products such as Horizon Worlds and Horizon Workrooms, where letting an avatar smile, raise an eyebrow, or make eye contact is intended to strengthen social presence, "the feeling that you're right there together with someone." Apple's Persona serves a similar purpose for telepresence-style video calls on Apple Vision Pro.

In the community-driven platform VRChat, face tracking is enabled through third-party tooling rather than a built-in feature. The open-source application VRCFaceTracking acts as a bridge between tracking hardware and VRChat, translating raw sensor output into the Open Sound Control (OSC) parameters that VRChat avatars expect. It works with a range of devices, including the Quest Pro (which exposes eye and face data through Meta's Face and Eye OpenXR extensions) and HTC's SRanipal-based trackers. Because VRChat avatars and trackers use many different shape conventions, VRCFaceTracking defines an open standard called Unified Expressions that is designed to be compatible with shapes from other standards such as ARKit/PerfectSync, SRanipal, and FACS. On the Quest Pro specifically, this tracking can be delivered only to the PC version of VRChat over a link connection, not to the standalone Quest build.

Other uses

Beyond avatars, facial expression data can feed user experience research and affective applications by providing an objective signal of expression during a session, and it can support more natural non-verbal communication in collaborative and training scenarios. Because the same blendshape data can drive any compatible character, face tracking is also used for virtual production and live performance, where a performer's expressions are mapped onto a digital character in real time.

Privacy considerations

Facial movement is sensitive biometric data: a face conveys identity, emotion, attention, and speech, so the capture and handling of facial data has drawn the same kind of scrutiny that surrounds eye tracking. The dominant industry response has been to process facial data on the device and avoid transmitting raw imagery. Meta states that Natural Facial Expressions is off by default, that the feature produces only a set of numbers (expression estimates) rather than storing pictures, and that the captured images of the face never leave the headset and are deleted after processing. Apple similarly says that the data used to build a Vision Pro Persona does not leave the device, although after a call a Persona may remain stored in encrypted form on the other participants' devices for up to 30 days. Even with on-device processing, the broader concern remains that detailed facial expression data could in principle reveal emotional or health-related information, which is why these features are typically opt-in and governed by per-application permissions.

See Also

• Eye tracking
• Hand tracking
• Avatar
• Social presence
• Social VR
• Facial Action Coding System
• Meta Quest Pro
• Apple Vision Pro
• PlayStation VR2
• VRChat
• Horizon Worlds
• ARKit
• Head-mounted display
• Computer vision
• Infrared
• Privacy
• Biometric data