Audio-driven facial animation

Audio-driven facial animation, also called speech-driven facial animation, is the automatic generation of a character's facial motion from recorded or live speech audio. Systems range from lip sync alone, the movement of the mouth to match the spoken sounds, to animation of the whole face, including the tongue, eyes and emotional expression. Viseme-based systems convert speech into a timed sequence of visemes, mouth shapes that correspond to groups of speech sounds, and blend between them. Learned systems train machine learning models on recordings of real faces to predict mesh vertex positions or rig controls directly from the audio.

Meta's documentation states that on devices without inward-facing cameras, such as the Meta Quest 2 and Meta Quest 3, face tracking relies on audio from the microphone to estimate facial movements, and its Movement SDK lists the Quest 2, Quest 3 and Meta Quest 3S as supporting "Audio to Expressions only". VRChat animates avatar mouths with visemes detected from the user's voice. A 2025 study of professional VTubers reports that wearing a headset prevents iPhone face tracking from working properly, so VTubers who perform in VR rely on microphones to synchronize their avatars' lip movements with their voice. The same technique animates characters whose speech is produced at runtime by text-to-speech, such as generative AI non-player characters.

Software for the task includes Oculus Lipsync, which Oculus released as a Unity plugin in February 2016; NVIDIA's Audio2Face, whose models, SDK, plugins and training framework NVIDIA open-sourced in September 2025; and the audio-driven mode of Epic Games' MetaHuman Animator, which Epic added as an experimental feature in Unreal Engine 5.5 in November 2024. Neural-network methods presented at SIGGRAPH 2017 by researchers from NVIDIA, Remedy Entertainment and Aalto University, and by a team from Disney Research and three universities, were followed by models based on transformers, learned codebooks of facial motion and diffusion.

Reviewed 1 November 2026. All 34 cited sources opened: nine papers and the Audio2Face-3D preprint in full text, Fisher (1968) by abstract, and the Meta, VRChat, NVIDIA, Epic Games, JALI Research and press pages, for dates, figures, quotes and attributions. About review dates.

Terminology

The term viseme, short for visible phoneme, goes back to a 1968 study by Cletus G. Fisher of the University of Iowa. In that study, 18 college students with normal hearing responded to the visual perception of initial and final consonants, and the significant confusions among the consonants were "grouped into mutually exclusive classes termed visemes". In facial animation the term refers to the shape of the mouth at the apex of a given phoneme. According to the JALI authors, procedural speech animation maps phonemes to visemes by rules or look-up tables, typically many-to-one: /m/, /b/ and /p/, for example, all map to the same viseme. Karras and colleagues note that in general the mapping between phonemes and visemes is many-to-many. The mapping is further complicated by coarticulation: in the JALI authors' words, when humans speak, visemes "overlap and crowd each other out in subtle ways", which makes the coarticulation scheme the crux of any procedural model.

Output usually takes one of two forms. Some systems set the controls of a face rig, such as blendshape (morph target) weights. According to the JALI authors, facial animation tools in industrial practice typically use software such as Maya to animate a 3D facial rig, often by simple interpolation between an array of target blend shapes, while more principled rigs combine a skeletally animated jaw with target shapes for the facial muscle action units of the Facial Action Coding System (FACS). The controls of JALI's own rig are based on FACS. Meta's Movement SDK can output either FACS-based expressions, which its documentation describes as representing 70 muscles used to animate the face, or 15 viseme blendshapes. Other systems predict the positions or displacements of the vertices of a face mesh with a fixed topology. NVIDIA's Audio2Face-3D paper notes that such vertex output is tied to a specific face identity, which limits its direct use where users want to animate their own characters. NVIDIA therefore provides a general blendshape solver that fits a blendshape model to the generated vertex motion, and an ARKit-compatible model for obtaining the corresponding weights, which face rigs such as Epic Games' MetaHumans can use.

How it works

Viseme-based and procedural methods

Viseme-based systems first establish which speech sounds occur and when. JALI, presented at SIGGRAPH 2016 by Pif Edwards, Chris Landreth, Eugene Fiume and Karan Singh of the University of Toronto, takes a speech transcript and the corresponding audio, computes a phonetic alignment with forced alignment, and uses the transcript and alignment to procedurally animate the phonemes on a JALI rig. Audio features such as volume, pitch and formants drive the JALI viseme field, which has separate dimensions for jaw and lip activity and whose values modulate the animation curves. All four authors are co-founders of JALI Research, which says it launched its first product in 2020 together with Cyberpunk 2077, with CD Projekt Red as development partner, and has since worked with dozens of teams. In a video breakdown reported by PCGamesN in October 2020, CD Projekt Red's Mateusz Popławski said that with Cyberpunk 2077 the team had set out to achieve "better quality lip-sync than The Witcher 3 with muscle-driven emotional expressions". PCGamesN identified JALI as the technology behind the work and reported that the system adapts animation elements such as facial inflections to each of the game's ten supported languages.

Other systems classify the audio directly, without a transcript. Oculus Lipsync analyzes a recorded or live audio stream, such as voice chat captured by a microphone, and produces values for 15 visemes (sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, ih, oh and ou) that animate an avatar's lips. Meta's documentation says these visemes were selected "to give the maximum range of lip movement" and "are agnostic to language", and that Lipsync version 1.30.0 and newer also detects laughter. In an August 2018 tech note, Sam Johnson, Colin Lea and Ronit Kassis wrote that the original model, a "small and shallow neural network" that mapped short segments of speech audio to phonemes, had worked reasonably well for English but not for other languages, and had not been robust to background noise. Oculus replaced it with temporal convolutional networks (TCNs). The work began at Facebook Reality Labs in PyTorch, and the model was converted to Caffe2 with ONNX for real-time processing. In internal testing the TCN models achieved "over 30% higher viseme accuracy on English speech" and significantly outperformed the earlier model on speech with heavy accents or large amounts of background noise. Some methods use only loudness: VRChat's "Jaw Flap" lip sync modes open an avatar's jaw bone, or animate a single mouth blend shape, according to how loudly the user speaks.

Learned methods

Learned methods replace hand-built rules with a model trained on synchronized recordings of speech and 3D face motion. At SIGGRAPH 2017, Tero Karras and colleagues from NVIDIA, Remedy Entertainment and Aalto University presented a convolutional neural network that maps 520 ms of audio, 260 ms before and after the output frame, to the 3D positions of the 5,022 vertices of a face mesh. It was trained on 3 to 5 minutes of performance-capture data per actor. Because the same sound can be produced with very different facial expressions, a network with nothing but the audio to work with "will learn to output the statistical mean" of conflicting training examples. The network therefore also learns, without predefined labels, a compact latent code for the apparent emotional state, which can be used as a control at inference time. The authors left the tongue, wrinkles, eyes and residual head and body motion to higher-level procedural animation controls. Taylor and colleagues, from the University of East Anglia, the California Institute of Technology, Disney Research and Carnegie Mellon University, used phoneme labels instead of raw audio. Their sliding-window deep neural network maps a phoneme sequence, obtained with off-the-shelf speech recognition or forced-alignment software, to a reference face model whose animation can be retargeted to any computer-generated character, and it can generate speech animation in real time from a user's speech.

Later systems generalized across speakers and faces. VOCA (Voice Operated Character Animation), from the Max Planck Institute for Intelligent Systems, extracts audio features with DeepSpeech, which its authors say makes it robust to different audio sources and noise, and outputs vertex displacements on the FLAME head model. It can synthesize different speaking styles and is "readily applicable to unseen subjects without retargeting". MeshTalk, from Facebook Reality Labs and Carnegie Mellon University, learns a categorical latent space that separates audio-correlated from audio-uncorrelated information, so that it produces accurate lip motion together with plausible motion of parts of the face that do not correlate with the audio, such as eye blinks and eyebrow motion. FaceFormer uses an autoregressive transformer whose audio encoder is initialized with the self-supervised pre-trained speech model wav2vec 2.0, to cope with the scarcity of 3D audio-visual training data. CodeTalker, from the Chinese University of Hong Kong and Tencent AI Lab, casts the task as a code query in a discrete codebook of facial motion, learned with a VQ-VAE by self-reconstruction over real facial motions; its authors say this makes the generated motion more vivid by reducing the uncertainty of the cross-modal mapping.

NVIDIA's Audio2Face-3D, described in a 2025 preprint, offers two kinds of network trained on the same data, recorded from professional actors with multiple machine vision cameras provided by the commercial facial capture service DI4D. Regression networks, released for three actors named Mark, Claire and James, estimate each facial pose from 0.52 seconds of audio and output one frame per inference; NVIDIA describes them as lighter on memory and as scaling well to many concurrent processes. A diffusion-based network supports multiple identities with a single network and, according to NVIDIA, generally produces higher-quality and more expressive animation. Both take audio and emotion labels as input and output motion for the facial skin, tongue, jaw and eyeballs, and an optional Audio2Emotion network can detect the emotion in the voice and supply the emotion input.

Real-time operation and latency

Live use requires low latency, but coarticulation means that a system has to hear some of the upcoming audio. According to Karras and colleagues, coarticulation sets a lower bound for this look-ahead. Their network's audio window reaches 260 ms into the future; the look-ahead could be limited to 100 ms during training with minor degradation in quality, but shortening it further "leads to a quick drop in perceived responsiveness", so they put a realistic lower bound for the latency of their method at around 100 ms. In their Theano implementation, inference took 6.3 ms for a single frame. The 2018 Oculus Lipsync model takes a stream of low-level audio features from the past as input and, in offline applications, can also use future audio. In an April 2019 blog post on the design of Oculus Avatars, Mike Howard wrote that the team had considered delaying the audio long enough to animate into the visemes, but rejected the idea because it would worsen the common frustrations of videoconferencing and long-distance calls. Instead, the team modeled the intermediate mouth shapes between sounds by controlling how quickly individual mouth muscles could move, a technique it nicknamed differential interpolation. The Audio2Face-3D diffusion network streams by processing a one-second audio window and generating the central half second of animation, advancing 0.5 seconds at a time, and NVIDIA reports that two diffusion steps are sufficient for high-quality output while significantly reducing inference time. With TensorRT, NVIDIA reports 1.4 GB of GPU memory for one track of this network and 4.0 GB for eight concurrent tracks, against 0.6 GB for eight tracks (and 1.3 GB for 128) of a regression network, and attributes the difference mainly to output size: per inference, the regression network outputs one frame of PCA-compressed data and the diffusion network 30 frames of raw vertex data.

History

According to the CodeTalker authors, early work on speech-driven animation mainly built complex mapping rules between phonemes and their visual counterparts. Karras and colleagues name the dominance model of Cohen and Massaro (1993) as a well-known example of such rule-based systems, and in their 2017 user study they compared their network against dominance-model animation produced with FaceFX, a commercial package they describe as widely used. Oculus released Oculus Lipsync for Unity in February 2016, and in 2019 Howard described its use to derive the mouth shapes of Oculus Avatars from spoken audio. JALI, published the same year as Lipsync, combined procedural viseme animation with a two-dimensional jaw and lip model driven by audio features.

The two SIGGRAPH 2017 papers trained deep neural networks on captured speech performances: Karras and colleagues mapped raw audio to mesh vertices, and Taylor and colleagues mapped phoneme labels to a retargetable reference face. Models that apply to unseen speakers (VOCA, 2019), also animate the upper face (MeshTalk, 2021) or use transformers (FaceFormer, 2022) followed. In April 2021 CG Channel reported that NVIDIA's Omniverse Audio2Face app was due to enter open beta at the end of the GTC 2021 conference, for Windows 10 and any RTX GPU, and that besides processing offline recordings of dialogue it would let users connect a microphone to drive a 3D character's face live. Epic Games added audio-driven animation to MetaHuman Animator in Unreal Engine 5.5 in November 2024, Meta introduced Audio To Expression in December 2024, and NVIDIA open-sourced the Audio2Face-3D models and tools in September 2025.

Products and SDKs

Product	Developer	Introduced	Approach and output	Status (November 2026)
Oculus Lipsync	Oculus (now Meta Platforms)	February 2016, as a Unity plugin	Analysis of recorded or live audio into values for 15 visemes; laughter detection from version 1.30.0	Meta's download page lists version 29.0.0 (24 May 2021), described as a maintenance release, as the latest; the Unity documentation is filed in the archive section of Meta's developer documentation
Audio To Expression (Movement SDK)	Meta	December 2024	Machine learning model trained on speech samples that translates the audio stream into FACS-based expressions or visemes, including expressions of the cheeks, eyes, eyebrows and nose	Documentation for Unity, Unreal Engine and native OpenXR apps; Meta's documentation lists Quest 2, Quest 3 and Quest 3S as audio-only and Quest Pro as visual and audio
Audio2Face (Audio2Face-3D)	NVIDIA	April 2021, open beta announced as an Omniverse app	Regression and diffusion networks for skin, tongue, jaw and eyeball motion, with ARKit blendshape output; Audio2Emotion network for emotion input	Models, SDK, plugins and training framework open-sourced on 24 September 2025; the Omniverse Launcher, through which the original app was installed, was deprecated on 1 October 2025
MetaHuman Animator audio-driven animation	Epic Games	November 2024, Unreal Engine 5.5, as an Experimental feature	Animation for all MetaHuman facial controls from audio, including inferred upper-face gestures	Real-time animation from audio input added with Unreal Engine 5.6; current documentation requires Unreal Engine 5.6 or later
JALI	JALI Research	2020, with Cyberpunk 2077	Jaw and lip viseme model driven by audio and a transcript	Offered under several pricing and licensing options

NVIDIA's open-source release includes the Audio2Face-3D SDK, a plugin for Autodesk Maya and a plugin for Unreal Engine 5.5 and 5.6, all under the MIT License, and the training framework under the Apache License. Its repository lists the pretrained Audio2Face-3D models under the "Nvidia Open Model" license, the Audio2Emotion models under a custom license that allows their use only with Audio2Face, and the sample training dataset for evaluation only; the Audio2Face-3D NIM container for large-scale deployment is covered by the NVIDIA Software License Agreement and product-specific terms for AI products. According to the repository, Audio2Face-3D can drive a character's face through direct mesh deformations, joint transformations or blend shape weights. NVIDIA states that Convai, Codemasters, GSC Games World, Inworld AI, NetEase, Reallusion, Perfect World Games, Streamlabs and UneeQ Digital Humans have integrated Audio2Face in their applications, and that The Farm 51 integrated it into Chernobylite 2: Exclusion Zone. Since the deprecation of the Omniverse Launcher, NVIDIA's legacy-tools page lists Audio2Face as existing within the Audio2Face-3D SDK, training framework, training dataset and microservices.

Epic Games' release notes for Unreal Engine 5.5 describe the MetaHuman Animator feature as delivering "animation for all MetaHuman facial controls, including inference of upper face gestures", as working "with various voices and languages", and as "A fully local, offline solution, suitable for batch-processing audio". In the 5.5 workflow, a SoundWave asset is processed into facial rig animation tracks in Sequencer, which can then be exported. The current documentation says processing can be done in real time or offline, with options to generate head movement and blinks, to limit output to mouth-region curves for swapping or layering lip animation, and to override the detected mood with settings such as happy, sad, fear or anger.

Use in VR

Avatars

In his 2019 post on the design of Oculus Avatars, Howard wrote that the team had to deliver "eye and mouth movements without any cameras". Mouth shapes were derived from spoken audio with Oculus Lipsync, and gaze was predicted from head direction and the objects or people in front of the user in the virtual environment. Because avatars with eye and mouth movements alone felt very inorganic, especially when they were not talking, the team added micro-expressions to its face model, such as slight tightening around the eyes and cheeks and periodic twitches around the mouth and brow, which Howard called "the base layer of what makes a face feel alive". He also wrote that the team needed to "tread incredibly carefully" in simulating when someone might look angry, or even happy, without any data about the social context, since adding a small degree of anger "invoked a visceral response in VR".

Meta's Movement SDK documentation lists Quest 2, Quest 3 and Quest 3S as supporting "Audio to Expressions only" and Meta Quest Pro as supporting "Visual and Audio to Expressions". In Meta's terms, visual estimation uses the inward-facing cameras and optional microphone data, while audio estimation uses microphone data only. Meta calls the feature Audio To Expression in its December 2024 announcement and Audio To Expressions in its documentation, which describes a machine learning model trained on speech samples that translates the audio stream into either FACS expressions or Oculus visemes. The announcement says that much of the face's expressivity comes from areas such as the cheeks, eyes, eyebrows and nose, whose subtle movements traditional viseme-driven blendshapes miss, and that Audio To Expression uses the app's audio stream "to produce full facial representations of people using only their voices", so that expressions become more meaningful while users are speaking, laughing or even coughing. Meta adds that the feature "requires just a fraction of the overhead compared to the earlier Lipsync library". In native OpenXR apps, audio is one of the data sources of the XR_FB_face_tracking2 extension, which Meta recommends over the deprecated XR_FB_face_tracking because the older extension does not support tongue tracking or audio-driven face tracking. Apps must request the RECORD_AUDIO permission, in addition to the face tracking permission, if facial movements are to be estimated from the microphone. For the Meta VR Glasses, which UploadVR reported would ship in spring 2027, Meta's documentation states that "There's no camera-based face tracking available to apps" and that face blendshapes for apps are driven by Audio To Expressions, "the same audio-driven approach that powers stylized avatars on Quest 3 and Quest 3S today".

In VRChat, avatar creators choose one of six lip sync modes in the VRC Avatar Descriptor component. The recommended "Viseme Blend Shape" mode drives the avatar's viseme blend shapes, and VRChat states that it uses "the Oculus Audio library to detect and set visemes". Another mode routes these audio-driven visemes through VRChat's face tracking system, which supports interaction with tracked mouth expressions. In their 2025 study of VTubing, for which they surveyed 18 professional VTubers and interviewed 16 of them, Kim and colleagues write that wearing a head-mounted display prevents the iPhone's face tracking from working properly and that, as a remedy, VTubers "rely on microphones to capture their speech and synchronize the avatar's lip movements with their voice".

Researchers have also named telepresence as an application. Karras and colleagues write that emerging real-time applications in telepresence and virtual reality avatars add challenges: no transcript, wide variability in user voices and physical setups, and "stringent latency requirements". The VOCA authors describe their model as suitable for tasks such as virtual reality avatars, in which the speaker, speech or language is not known in advance. The MeshTalk authors hoped their approach would be a stepping stone towards VR telepresence, because as head-mounted capture devices become smaller, regressing accurate lip and tongue motion from their oblique camera viewpoints becomes harder.

Non-player characters

For characters whose speech is synthesized at runtime, the generated audio is the input for the face. The Audio2Face-3D paper describes the system as particularly efficient in two domains, games and AI-driven interactive avatars. It notes that games "often require hundreds of hours of facial animation", which audio-driven animation can produce more efficiently than video capture, and that interactive avatars "typically receive only audio input" from large language models and text-to-speech systems, so their facial animation has to be generated at an interactive rate. NVIDIA says the animation data can be rendered offline for pre-scripted content or streamed in real time for dynamic, AI-driven characters. Photorealistic conversational characters of this kind are described in Digital human.

Survios used Audio2Face in Alien: Rogue Incursion Evolved Edition, a flatscreen version of its VR game Alien: Rogue Incursion for PC and PlayStation 5 that the studio announced in May 2025. Eugene Elkin, game director and lead engineer at Survios, said in NVIDIA's announcement that integrating Audio2Face had "streamlined the pipeline for lip-syncing and facial capture".

Research

Papers evaluate audio-driven animation with objective error measures and with perceptual studies. MeshTalk defines the lip error of a frame as the maximal L2 error of all lip vertices and reports the average over all frames of the test set, because the average over all lip vertices tends to mask inaccurate lip shapes while the maximal error correlates better with perceptual quality. FaceFormer and CodeTalker use the same lip vertex error. The same baseline can receive different values in different papers: on the BIWI-Test-A split, FaceFormer's paper reports a lip vertex error of 7.6427 for VOCA and CodeTalker's paper 6.5563 (both in units of 10^-4 mm). FaceFormer's authors note that, given the many-to-many mappings between upper-face motion and speech, qualitative evaluation and user studies have been suggested as more appropriate than quantitative metrics for evaluating speech-driven facial animation. The Audio2Face-3D paper uses a pretrained SyncNet lip-sync score, a jitter measure, a score for bilabial sounds and an expressiveness score, but states that it uses this benchmark primarily for sanity checking and that the main evaluation of network quality is subjective assessment by experts.

Method	Venue	Input	Output	Training data	Reported results
Karras et al.	SIGGRAPH 2017 (ACM TOG)	520 ms audio window	Positions of 5,022 mesh vertices	Vision-based capture with nine video cameras (DI4D PRO): 5 min 1 s for one character and 3 min 45 s for another	In a blind study with 20 participants (260 pairwise votes per comparison), preferred to FaceFX dominance-model animation in 87% of votes; video-based performance capture was preferred to it in 77%
Taylor et al.	SIGGRAPH 2017 (ACM TOG)	Phoneme labels	Reference face model, retargeted to character rigs	KB-2k: one actor reciting 2,543 phonetically diverse TIMIT sentences	Demonstrated on a variety of characters and voices, including singing and foreign-language input; real-time generation from user speech
VOCA	CVPR 2019	DeepSpeech audio features and a speaking-style condition	Vertex displacements on the FLAME template (5,023 vertices)	VOCASET: 12 subjects, 480 sequences, about 29 minutes of 4D scans at 60 fps	Applies to unseen subjects without retargeting; dataset and model released for research purposes
MeshTalk	ICCV 2021	Audio, with a categorical latent space for expression	Full face mesh (6,172 vertices)	In-house: 250 subjects reading 50 sentences each, tracked from 80 cameras at 30 fps; 13 hours in total	Lip error on the in-house test data of 3.184 mm versus 3.720 mm for VOCA; with 100 participants, rated better than or equal to VOCA in 75.3% of full-face comparisons; against tracked ground truth, 42.1% of full-face pairs favored the ground truth, 35.7% were rated equal and 22.2% favored MeshTalk
FaceFormer	CVPR 2022	Raw audio (wav2vec 2.0 encoder)	Mesh vertex motion	BIWI and VOCASET	Lip vertex error on BIWI-Test-A of 5.3742, against 6.7436 for MeshTalk and 7.6427 for VOCA (10-4 mm); in Amazon Mechanical Turk A/B tests on BIWI-Test-B, preferred for realism to VOCA in 83.85% and to the ground truth in 35.24% of answers
CodeTalker	CVPR 2023	Speech audio	Facial motion decoded from a learned discrete codebook	BIWI and VOCASET	Lip vertex error on BIWI-Test-A of 4.7914, against 5.3077 for FaceFormer in the authors' comparison (10-4 mm); user study reported
Audio2Face-3D	arXiv preprint, 2025	Audio and emotion labels	Skin, tongue, jaw and eyeball motion; ARKit blendshape weights after a blendshape solve	Professional actors recorded with machine vision cameras provided by DI4D, each performing about 50 to 70 sentences chosen to cover most English or Mandarin pronunciations, in 11 emotional states	Metric benchmark used for sanity checks; quality judged mainly by expert subjective assessment

Research in VR has examined how lip sync affects embodiment of an avatar. In a 2020 study in IEEE Transactions on Visualization and Computer Graphics, Mar Gonzalez-Franco, Anthony Steed, Steve Hoogendyk and Eyal Ofek had 20 participants (four of them female, aged 27 to 51) wearing an HTC Vive embody a gender-matched avatar that they saw in a virtual mirror and that neither looked like them nor was photorealistic. There were three conditions: a still face; lip sync, in which a lip-sync engine captured the participant's speech from the microphone and moved the avatar's lips in real time; and lip sync combined with ambient facial animations, such as blinks, designed by a professional animator. In a face-morphing self-recognition task, both lip-sync conditions led participants to recognize themselves earlier in the avatar-to-self morph than the still face did (p = 0.01). In the self-to-avatar direction, participants in the animated condition took longer than with lip sync alone to stop the morph (p = 0.029), which the authors interpret as a greater enfacement illusion. Participants looked at the avatar's face more in both lip-sync conditions than with the still face (p < 0.0001), and reported stronger embodiment in both lip-sync conditions than with the still face on the questionnaire statement "I felt as if the virtual body I saw when I looked down was my body" (p = 0.006). The authors report that the other questions, and the aggregated score, did not show significant differences.

Limitations

Speech audio carries little information about parts of the face that are not involved in producing sound. Karras and colleagues write that the eyes and eyebrows "have no direct causal relationship with sound production", and list as a limitation that their method cannot model blinking and eye motion correctly, since these do not correlate with the audio. The MeshTalk authors describe existing audio-driven methods as exhibiting "uncanny or static upper face animation". NVIDIA states that Audio2Face-3D "may struggle with generating semantically meaningful upper-face or eyeball motion"; because the system processes audio from a short temporal window, its upper-face and eyeball animations appear simple and fail to capture the semantic context. NVIDIA adds that it cannot generate natural idle motion or listening motion from long silent audio or from a conversation partner's speech. Meta's documentation lists as known issues that yawning may not open a character's mouth "as the models are trained mostly on verbal input", and that when an app first launches it may take up to 10 seconds before the voice fully drives the face animation. It also notes that the model is trained on how users really move their lips, and that for users who expect more exaggerated movements, like those provided by visemes, developers can increase the multipliers in the retargeting configuration file.

Recording conditions affect the result. Epic's documentation for MetaHuman Animator says that animation quality depends on the clarity of the recorded audio, recommends a sampling rate of at least 16 kHz and a high signal-to-noise ratio, advises against reverb and echo, and notes that the feature animates all voices present in a recording, including background voices. NVIDIA lists severe background noise, strong non-verbal sounds and lip shapes beyond the training data as cases in which Audio2Face-3D may fail to generate accurate lip movements. Systems that work from a transcript depend on its accuracy: the JALI authors write that extra, missing or mispronounced words and punctuation can result in poor alignment and cascading errors in the animated speech.

The CodeTalker authors attribute the remaining gap in realism and vividness to the highly ill-posed nature of the task and the scarcity of audio-visual data, and state that regression formulations suffer from "the regression-to-mean problem leading to over-smoothed facial motions". Some research models are not designed for live use: FaceFormer's authors describe the quadratic memory and time complexity of its self-attention mechanism as making the model "not suitable for real-time applications". Karras and colleagues note that almost but not quite human-like appearance and motion is often perceived as creepy, the effect known as the uncanny valley, and that other authors have tried to lessen it by deducing head movements, eye saccades or blinks from audio. Their own method assumes that such residual motion is driven by higher-level procedural controls, such as a game engine.

Learned models depend on captured performances of real people. The FaceFormer authors write that it is important to obtain consent from the actors during data acquisition, and that because their method can animate a realistic 3D talking face from an arbitrary audio signal, such techniques "could potentially be misused to cause embarrassment".

See also

• Facial motion capture
• Markerless motion capture
• Full-body pose estimation from sparse tracking
• Codec Avatars
• How to develop VR games with AI