Toggle search
Toggle menu
5.5K
6.1K
231
53.7K
VR & AR Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Generative AI non-player characters

vrarwiki.com · VR & AR Wiki
More actions

Generative AI non-player characters (also called generative AI NPCs, AI NPCs or LLM-powered NPCs) are characters in video games and virtual worlds whose dialogue, and in some systems whose actions, are produced by generative AI models while the game runs, instead of being selected from lines and branches written in advance. Voice-driven systems built for virtual reality (VR) chain several models: speech recognition turns the player's voice into text, a large language model (LLM) writes a reply from a description of the character and the conversation so far, a text-to-speech model voices the reply, and in some systems the character's mouth is animated to match.[1][2][3]

In VR, players can talk to these characters aloud: in a 2026 study of a VR game, the headset's microphone captured players' speech and the character platform's built-in speech recognition transcribed it.[4] A 2024 analysis of 20 popular VR games found that, where players held a dialogue with an NPC, the most common method was choosing what to say from a menu of dialogue options (6 of the 20 games), and that in the only game with voice commands, Phasmophobia, the NPC could recognize 55 predefined phrases.[2] Documented uses in VR and augmented reality (AR) include the conversational NPCs that Meta opened to all Horizon Worlds creators in September 2025,[5] the Meta Quest game Stellar Cafe (December 2025),[6] the community mod Mantella, which works in Skyrim VR,[7] and the virtual creatures of Niantic's AR game Peridot, whose reactions Niantic generated with a custom version of Meta's Llama 2 model.[8]

In studies of VR prototypes, mean waits for a character's spoken answer have ranged from about 1.5 seconds to about 6.9 seconds.[3][1] A study that deliberately delayed the characters' answers found that waits above 4 seconds degraded the quality of experience.[3] In a 2026 study, 13 of 48 players reported characters giving incorrect or made-up information.[4] Commercial use has raised further issues: moderation, after players led an AI-voiced Darth Vader in Fortnite into offensive replies in May 2025;[9] store rules such as Valve's disclosure requirements for "Live-Generated" AI content on Steam;[10] and the consent and pay of performers whose voices are replicated, which the SAG-AFTRA video game agreement approved in July 2025 addresses.[11]

Reviewed 1 November 2026. Checked all nine papers (full text or abstract, DOIs via Crossref) and every cited vendor, store, union and press source for the facts, figures, quotes and dates stated, plus links, categories and style. About review dates.

Background

Traditional decision-making methods for non-player characters include state machines and behavior trees; according to a 2025 preprint by researchers at the University of Oulu, these methods are increasingly supplemented or replaced by more sophisticated techniques, including machine learning and procedural content generation.[1] Zargham and colleagues write that social interaction in single-player games is either completely absent or limited to scripted interactions with NPCs that mostly rely on dialogue trees.[4] Christiansen and colleagues, authors of the 2024 analysis of VR games, write that traditional dialogue systems rely on time-consuming and non-scalable human-written scripts, which can break the illusion of reality, particularly when NPCs fail to recognize players with whom they have previously interacted.[2]

A 2024 survey of large language models in games defines non-player characters as "agents which exist in virtual game worlds but whose actions are not directly controlled by the players"; they may serve as pets, allies, enemies, merchants, quest givers or bystanders.[12] The survey identifies two ways in which an LLM can control an NPC: through its dialogue and through its behavior, meaning the selection of in-game actions. It separates foreground NPCs, which form part of the game's narrative and whose dialogue is heavily constrained by the narrative and their role in it, from background NPCs, whose purpose is to make the environment more believable and whose dialogue is essentially small talk.[12]

How it works

A voice-driven implementation runs as a loop. In a VR interrogation game built in Unity for Meta Quest 2 by the Oulu researchers, the player's question was recorded as an audio clip and converted to text, the text was sent with the full chat history to OpenAI's GPT-4 Turbo model, and the reply was converted into speech for one of two characters; speech-to-text, text-to-speech and the language model were implemented in Unity through the OpenAI API.[1] The 2024 study by Christiansen and colleagues used the Whisper API for transcription and sent each new message, the message history and a character backstory prompt to the gpt-4-1106-preview model, with the Oculus Voice SDK producing speech.[2] A study presented at the 2025 ACM Conference on Conversational User Interfaces (CUI) used a local server running Faster-Whisper for speech recognition and a quantized Llama 3.1 8B model on an RTX 4090 graphics card, with Edge-TTS for speech and OVR Lipsync for mouth movement.[3] Lip sync and facial expression generated from speech audio are covered in Audio-driven facial animation, and photorealistic characters in Digital human.

Commercial toolkits package the same stages. NVIDIA announced its Avatar Cloud Engine (ACE) for Games in May 2023 as a custom AI model foundry service whose foundation models for speech, conversation and character animation included NeMo, for building, customizing and deploying language models; Riva, for automatic speech recognition and text-to-speech; and Omniverse Audio2Face, for facial animation.[13] The ACE microservices it made generally available in June 2024 included Riva speech recognition, text-to-speech and translation, Nemotron language models, Audio2Face and Omniverse RTX rendering.[14] Inworld AI describes its Inworld Runtime as a "C++ graph engine", with SDKs such as Node.js and Unreal, that orchestrates LLMs, speech-to-text, text-to-speech, memory or knowledge, and tools in a single pipeline.[15]

Character definition and game context

The character is defined in text that accompanies each request to the model. On Inworld's platform, which the 2026 study used to build its game, each NPC is defined by a "Core Description" of its motivations, flaws and actions and can be given attributes such as name, pronouns, role and interests; the study's authors describe these settings as persistent character prompts that guide the LLM, and note that the platform abstracts away underlying components such as the specific LLM architecture.[4] Meta's Character Builder for Horizon Worlds takes traits such as a name, backstory, instructions and guardrails.[5] NVIDIA wrote in 2023 that the language models in ACE "can be customized with lore and character backstories".[13] For its NEO NPCs prototype, Ubisoft said that the creative work needed for the demo's narrative design went "far beyond" what traditionally scripted NPC interactions usually require, with every detail, from personalities and backstories to agendas and emotions, crafted and programmed into the model; Inworld's technology let the narrative team build a background, knowledge base and conversational style for each NPC.[16]

Because the model improvises, its output may not fit the intended scenario. The 2024 survey says that LLMs perform "something like a theatrical improvisation" rather than acting as an actor studying a part, and that giving the model the current state of the environment (the items in it and their affordances, and the other characters and their actions) in addition to the conversation history can mitigate this.[12] According to UploadVR, Horizon Worlds creators can use TypeScript to give a character background context about the world state and knowledge of events, such as the player picking something up or a door opening, and can script "elicitations" that prompt the character to speak by describing its motivation.[5] The choice of model also affects the character: in the Oulu prototype, GPT-3.5 Turbo made the suspect, written as an Australian exchange student, use Australian slang to an extent that hindered comprehension, and the team moved to GPT-4 Turbo.[1]

Memory is handled outside the model. In the Oulu prototype, all previous questions and replies were sent again with each new question so that the model knew what had been said.[1] The generative agents of Park et al. store a "memory stream" of observations in natural language, retrieve entries by recency, importance and relevance, and periodically synthesize them into higher-level reflections.[17]

Some systems let the model choose actions as well as words. According to Meta, Niantic used a custom version of Llama 2 in Peridot to generate each creature's reaction in real time, selecting responses from a library of animations, and improved response quality by defining an expected response format in JSON.[8] NVIDIA's January 2025 material describes ACE "autonomous game characters" that "perceive, plan, and act like human players", such as PUBG Ally in PUBG: Battlegrounds, which uses the Mistral-Nemo-Minitron-8B-128k-instruct small language model.[18]

Cloud and on-device processing

The models can run on remote servers, on the player's PC or on the headset. NVIDIA said in 2023 that developers could deploy ACE models through its DGX Cloud, on GeForce RTX PCs or on premises.[13] In June 2024 the ACE microservices became generally available for cloud deployment, while versions for RTX AI PCs, including the Nemotron-3 4.5B small language model built to run on the device, were in early access.[14] At CES 2025 Meaning Machine's game Dead Meat was shown generating dialogue locally on a GeForce RTX 50 Series GPU; according to NVIDIA, the game had previously used LLMs in the cloud.[18] In November 2026 NVIDIA's ACE for Games page described ACE as providing "cloud and on-device AI models for speech, intelligence, and animation". It offered MIT-licensed Unreal Engine 5 plugins for speech recognition, a language model, text-to-speech and Audio2Face-3D facial animation, an In-Game Inferencing SDK for running models locally inside a game, and a Game Agent SDK, in beta, for connecting NPCs to game state, contextual knowledge and model-driven actions.[19]

UploadVR reported in September 2025 that Horizon Worlds NPCs are powered by Meta's Llama 4 model. In its own analysis, the publication wrote that high-quality responses require powerful server-side chips accessed through an API billed per token, and that on-device models would take so much compute that they would significantly affect rendering of the virtual world and take longer to respond.[5]

Meta's AI Building Blocks for Unity (see AI Building Blocks), in documentation last updated in July 2026, comprise blocks for object detection, large language models, speech-to-text and text-to-speech; each block's provider defines whether inference runs "in the cloud", "on a local machine" or "on the Meta Quest device". Meta's recommended setup is a Meta Quest 3 or Meta Quest 3S, Unity 6 or newer and Meta XR Core SDK v83 or later, and the page does not describe them as an NPC system.[20] The 2024 survey states that running AAA games and LLMs in parallel on consumer hardware is infeasible because of their computation requirements, so developers would have to host models on their own servers or use APIs; the cost of querying is recurring, rises as more players use the LLM, and cannot be properly estimated beforehand.[12]

Latency

Each stage adds to the wait between the end of the player's speech and the character's answer. In the 2024 study by Christiansen and colleagues, the authors timed 20 questions in each condition within the Unity environment: replies averaged 4.00 seconds (SD 0.30) with speech recognition and 2.66 seconds (SD 0.45) with dialogue options, a difference they attribute mainly to the transcription step.[2] The Oulu prototype averaged 6,909 ms per question-and-answer cycle (median 6,066 ms), with a minimum of 2,021 ms and a maximum of 24,362 ms; the game did not allow any API call to last longer than 15 seconds. The GPT-4 Turbo call was the slowest stage on average (3,113 ms, against 2,376 ms for text-to-speech and 1,365 ms for speech-to-text), and its latency was also affected by the growing size of the chat history; input length affected every stage, especially text-to-speech.[1]

The CUI 2025 study measured 1.47 seconds on average for its pipeline (0.13 s speech recognition, 0.47 s LLM, 0.87 s speech synthesis), then compared response delays of 1.5, 4.0 and 6.5 seconds with 54 participants wearing a Meta Quest Pro connected to a PC. The authors report that latency above 4 seconds degrades quality of experience and that natural conversational fillers, a gesture with a spoken filler line, improve perceived response time, especially at high delays. An artificial filler (a processing icon with a sound effect) was not significantly different from no filler, and the improvements from natural fillers over no filler on broader measures such as engagement, good impression, competence and willingness to interact again were not significant.[3] NVIDIA stated in 2023 that its ACE models "are optimized for latency".[13]

History

Date Event
28 May 2023 NVIDIA announces ACE for Games at Computex. In the Kairos demo, made with Convai and rendered in Unreal Engine 5, players put natural-language questions to Jin, an NPC who runs a ramen shop.[13]
October 2023 Park et al.'s "Generative Agents" paper, with 25 LLM-driven characters in a sandbox town inspired by The Sims, is published in the UIST '23 proceedings.[17]
6 November 2023 Xbox and Inworld AI announce a multi-year partnership to develop an AI design copilot and an AI character runtime engine, integrated into the game client, that would generate stories, quests and dialogue dynamically.[21]
November 2023 Niantic integrates Llama 2 into Peridot (reported by Meta in May 2024).[8]
January 2024 Valve publishes rules for games that use AI on Steam, separating Pre-Generated from Live-Generated content.[10]
18 March 2024 At GDC, NVIDIA presents Covert Protocol, a technology demonstration made by Inworld AI in partnership with NVIDIA, using Riva speech-to-text and Audio2Face.[22]
19 March 2024 Ubisoft shows its NEO NPCs prototype in a private demo at the Game Developers Conference, built with Inworld AI's Character Engine and LLM technology and NVIDIA Audio2Face.[16]
2 June 2024 NVIDIA makes the ACE microservices generally available for cloud deployment; an updated Covert Protocol demo, developed with Inworld AI and shown at Computex, runs Audio2Face and Riva speech recognition locally on GeForce RTX PCs.[14]
6 January 2025 NVIDIA announces ACE autonomous game characters at CES 2025, including PUBG Ally and the Smart Zoi characters of inZOI.[18]
May 2025 Players lead an AI-voiced Darth Vader in Fortnite into offensive replies; SAG-AFTRA files an unfair labor practice charge over the voice.[9]
9 July 2025 SAG-AFTRA members approve a new Interactive Media Agreement, ending the union's strike against video game companies.[11]
13 August 2025 Inworld AI announces Inworld Runtime.[23]
10 September 2025 Conversational LLM NPCs become available to all Horizon Worlds creators.[5]
21 November 2025 Ubisoft reveals Teammates, a playable experiment with an AI voice assistant and AI squadmates that respond to voice commands.[24]
11 December 2025 Stellar Cafe is released for Meta Quest headsets.[6]

Ubisoft's NEO NPCs demo had players talk to the characters by voice ("voice-to-chat") to reach their objectives; Ubisoft said the characters adapt emotionally to the player's intentions, react to points of interest in the environment and can generate new quests on the fly.[16] Its Teammates experiment, a short level built around the mechanics of a first-person shooter and shared with "a few hundred players in a closed playtest", has an AI voice assistant called Jaspar and two squad members, Pablo and Sofia, that generate "real-time responses and actions based on player voice commands"; the announcement does not name the models used.[24] GamesBeat reported that Inworld AI started with a focus on AI tools for game developers, such as smart non-player characters, and that it presented Runtime in August 2025 as an AI runtime engineered to scale consumer applications;[23] in October 2025 the company wrote that lifelike, interactive characters for games and entertainment remained "a core focus area".[15] In November 2026 the home page of Inworld's developer documentation offered two products: Voice, covering text-to-speech, speech-to-text and a realtime API for voice agents, and Inference, for text, image, video and decision models.[25]

Products and SDKs

The table lists tools documented for building such characters, with status as reported by the cited sources.

Product Developer Documented components Engines and platforms Status (dated)
ACE (Avatar Cloud Engine) NVIDIA Riva speech recognition, text-to-speech and translation; Nemotron language models; Audio2Face facial animation; Omniverse RTX rendering[14] Deployment in the cloud, on GeForce RTX PCs or on premises; Audio2Face connectors for Unreal Engine 5 (2023);[13] Unreal Engine 5 plugins and C/C++ SDKs for custom engines (page read November 2026)[19] Cloud microservices generally available since June 2024, when RTX AI PC versions were in early access;[14] in November 2026 NVIDIA offered on-device models, an In-Game Inferencing SDK and a Game Agent SDK in beta[19]
Inworld Runtime Inworld AI Graph engine orchestrating LLMs, speech-to-text, text-to-speech, memory or knowledge, and tools[15] C++ engine with SDKs such as Node.js and Unreal;[15] Unity AI Runtime SDK with a Character template[26] Announced August 2025.[23] The legacy Character Studio "has been retired", and its workflows have moved to the Inworld Portal and the Inworld Agent Runtime (undated notice, read November 2026)[26]
Convai Unity SDK Convai Voice conversation, lip sync, actions, emotion and long-term memory[27] Android-based XR headsets (Meta Quest, Horizon OS, Android XR) and Windows XR headsets; the built-in camera frame source for vision works only on Quest 3 and 3S, and other headsets need a custom frame source[27] Documentation read November 2026[27]
Horizon Worlds NPCs and Character Builder Meta Llama 4 language model, dozens of synthetic voices, TypeScript scripting[5] VR and mobile worlds in Horizon Worlds[28] Available to all creators since 10 September 2025; at launch only adult users in the US could talk to the characters.[5] In March 2026 Meta said it would keep existing Horizon Worlds games working in VR but bring no new games to VR[29]
AI Building Blocks Meta Object detection, LLM, speech-to-text and text-to-speech blocks[20] Unity 6 or newer; Meta Quest 3 or 3S; Meta XR Core SDK v83 or later[20] Documentation last updated July 2026[20]
CUIfy Buldu et al. (academic) Open-source Unity package connecting LLM, speech-to-text and text-to-speech models for speech-based NPCs; several NPCs per environment; streaming between models to reduce latency[30] XR applications built in Unity[30] Published at IEEE AIxVR 2025[30]
Mantella Art from the Machine (community project) Speech-to-text (Moonshine or Whisper), an LLM, text-to-speech (Piper, xVASynth or XTTS)[31] Skyrim and Fallout 4; the documentation lists "Fully playable in VR / SE / AE"[32] Open source under the AGPL-3.0 license; latest GitHub release v0.14 (21 April 2026), checked November 2026[31]

Use in VR and AR

Horizon Worlds

Meta previewed AI NPCs for Horizon Worlds at Meta Connect 2024 and made them available to selected Meta-partnered creators earlier in 2025.[5] On 27 August 2025 its developer blog said that "fully-embodied conversational LLM NPCs" were coming soon: creators would set a character's name, story, personality and dialogue with an expanded Character Builder, select voices from a library that can speak scripted and unscripted dialogue, combine scripted responses with LLM responses, and use the characters in both VR and mobile worlds.[28] According to Road to VR, Meta said such characters could serve as quest givers, guides, lore-deepening characters, shopkeepers or bosses that react dynamically.[33]

UploadVR reported on 10 September 2025, a week before Connect 2025, that all creators could add the characters, with "dozens of synthetic voices" adjustable in speed and pitch. Only adult users in the US could talk to them; for other players, scripts can check availability and fall back to direct text-to-speech that bypasses the LLM. Meta said it would add the ability for the characters to "trigger in-world actions" later in 2025.[5] In a recap of Connect 2025 published on 18 September 2025, Meta said that its NPC generation system could come up with a backstory, voice, conversation style and dialogue from a few prompts, called the world Citadel "one of the most popular worlds in Horizon", and stated that time spent in it "increased by nearly 15%" after its scripted NPCs were replaced with AI-generated characters.[34] In March 2026, days after Meta announced that Horizon Worlds would drop VR support in June, its chief technology officer, Andrew Bosworth, said that the company would "keep Horizon Worlds working in VR for existing games" but was not bringing new games, with most of its energy going towards mobile and the Meta Horizon Engine.[29]

Stellar Cafe

Stellar Cafe is a VR game built around spoken conversation with robot characters. Its developer, AstroBeam, was founded by Devin Reimer, a co-founder and former chief executive of Owlchemy Labs, the studio behind Job Simulator. Announcing the game on 12 August 2025, AstroBeam said that large language models interpret the player's voice input and generate the characters' voice output in real time, and that it had licensed Owlchemy Labs' hand-tracking technology for navigation; Reimer described the approach as "combining handcrafted characters with real-time AI conversation".[35] Road to VR's hands-on, published the same day, reported that the characters were "currently based on Google Gemini", although the studio said that could change, and that they build memories across sessions, such as the player's name or past choices; the writer found that the AI "can misunderstand a word, loop around sometimes in the middle of conversations".[36]

The game was released on 11 December 2025 for Meta Quest 2, 3 and 3S at US$14.99 (price as of UploadVR's January 2026 review). The reviewer found that responses differed across two playthroughs of the opening scenarios and that the voice interactions all worked; movement between locations is also controlled by voice, the game supports controllers or hand tracking, and its full runtime is about two hours. Looking around during a conversation was sometimes registered as a head shake, forcing the reviewer to repeat a prompt.[6]

Mantella and Skyrim VR

Mantella is an open-source mod that lets players speak naturally to the NPCs of The Elder Scrolls V: Skyrim and Fallout 4.[31] Its documentation lists "Fully playable in VR / SE / AE" among its features and says it works with local models such as Llama and Gemma or with APIs from OpenAI, OpenRouter and NanoGPT.[32] In a March 2025 test in Skyrim VR, UploadVR found that NPCs remembered earlier conversations, were aware of in-game context such as the time of day and could notice some of the items the player carried. A free setup through OpenRouter produced "significant lag time" in spoken replies; with a paid API key and the GPT-4o mini model the wait became much shorter, but conversations could still get stuck, or NPCs failed to respond, especially when several AI-driven characters were together in a crowd.[7]

Peridot and Project Jade

In Peridot the generative model drives reactions and animation rather than speech. Meta's May 2024 account says Niantic integrated Llama 2 in November 2023, and that the model takes into account each creature's traits, such as personality and history, and scene context such as hunger, attention and detected objects when it selects a response from the animation library.[8] At Snap's Lens Fest on 16 October 2025, Niantic Spatial showed Project Jade, a demonstration of Dot as an AI companion on Snap's Spectacles AR glasses. Using the company's Visual Positioning System and pathfinding maps generated algorithmically from 2D floor plans, Dot answers attendees' questions about sights and featured experiences and guides them there; its conversational voice comes from Hume AI's EVI.[37] On 23 April 2026 Niantic Spatial announced that it was winding down the Peridot mobile game: in-app purchases were disabled that day, the app was to be removed from the App Store and Google Play on 14 May 2026, and the servers were to close on 31 August 2026. The post pointed players to Hello, Dot (for Meta Quest 3 and Apple Vision Pro), Peridot Beyond (for Spectacles) and Project Jade.[38]

Other examples

Wisp World, an Apple Vision Pro app from Liquid City, is described in its App Store listing as the home of "a tiny forest spirit powered by advanced AI" that the player feeds and holds conversations with to help the plants grow; the listing does not identify the AI models. It cost US$9.99 as of November 2026.[39] Directed by Jasper Stevens and Keiichi Matsuda, it was nominated in the Best Game category at Raindance Immersive 2024.[40]

Research

Studies with prototype characters have measured response times, social presence, believability and usability, most of them in VR. Korkiakoski et al. is an arXiv preprint; the other studies below appeared in conference proceedings or a journal.

Study Venue Setup Participants Reported results
Park et al. (2023)[17] UIST '23 25 agents in a sprite-based sandbox town ("Smallville") running on gpt-3.5-turbo, with memory stream, reflection and planning; not VR 100 evaluators comparing interview answers (within-subjects) The full architecture produced the most believable behavior (TrueSkill 29.89, against 22.95 for answers written by human crowdworkers); news of a party spread from 1 to 13 agents and 5 of 12 invited agents attended; the simulation cost "thousands of dollars in token credits" and took multiple days
Wan et al. (2024)[41] CHI EA '24 (extended abstract) GPT-4 agent deployed as an NPC in VRChat, replying with context-relevant responses followed by facial expressions and body gestures Preliminary evaluation Identified the parameters that gave the "most plausible responses"
Christiansen et al. (2024)[2] VRST '24 VR murder mystery in Unity on Quest 2 (laptop with an RTX 3060 GPU); free speech or dialogue options, with GPT-4 answering in both 24 (22 analyzed), between-subjects No statistically significant difference in social presence; speech felt more immersive but "overwhelming"; replies took 4.00 s versus 2.66 s, a difference the authors describe as a possible confounding variable; 8 of the 13 correct guesses of the murderer came from the dialogue-option group, not a statistically significant association
Buldu et al. (2025)[30] IEEE AIxVR 2025 CUIfy, an open-source, privacy-aware Unity package for LLM-powered NPCs with speech input and output System paper Supports several NPCs per environment; streams between models to reduce latency
Maslych et al. (2025)[3] CUI '25 Meta Quest Pro; local Llama 3.1 8B and Faster-Whisper, Edge-TTS; delays of 1.5, 4.0 and 6.5 s, with no filler, an artificial filler (icon and sound) or a natural filler (gesture and spoken line) 54, within-subjects Delays above 4 s degraded quality of experience; natural fillers improved perceived response time, mainly at high delay; artificial fillers did not differ significantly from none
Korkiakoski et al. (2025)[1] arXiv preprint VR interrogation simulator in Unity on Quest 2 with a suspect and a partner NPC driven by GPT-4 Turbo 18 (convenience sample, aged 20-25) Mean cycle latency about 7 s; believability 6.67 out of 10, with high ratings for behavior, social relationships and intelligence and moderate ratings for emotion and personality; System Usability Scale score 79.44
Zargham et al. (2026)[4] International Journal of Human-Computer Interaction Office Whispers, a VR adventure-puzzle game with four NPCs built on Inworld's platform in Unity, on Quest 2 48 (in Canada and the Netherlands) Overall experience rated 5.75 out of 7; 37 participants (77%) reported speech recognition problems; 13 (27%) said NPCs sometimes gave incorrect information

Two reviews summarize the wider field. The 2024 survey by Gallotta and colleagues in IEEE Transactions on Games covers the roles LLMs can take in games, NPCs among them, and their limitations.[12] A scoping review in IEEE Access by Süeda Özkaya and colleagues, drawing on 66 peer-reviewed studies published between 2018 and 2025, discusses "critical challenges" for LLMs in VR games, including real-time performance constraints, memory limitations, ethical risks and scalability barriers.[42]

Limitations

LLMs produce hallucinations, which the 2024 survey describes as plausible but false statements; it gives the example of NPCs that invent quests which do not exist in the game.[12] In the Office Whispers study, 13 of 48 players said NPCs sometimes provided incorrect information or "hallucinated" facts that hindered their progress, and 14 were frustrated by generic replies such as "Sorry, I don't know about that!"[4] Christiansen and colleagues observed a tendency to hallucinate answers when the model was given too little information,[2] and in the generative agents evaluation the most common errors arose when agents failed to retrieve relevant memories, fabricated embellishments to their memories or inherited overly formal speech or behavior from the language model.[17]

Speech input adds its own failures. In the 2026 study, 37 of 48 participants reported speech recognition problems, and 24 of them attributed the problems to their own unclear speech or accents; five said NPCs interrupted them, and four described extra effort such as enunciating, speaking louder or slowing down.[4] Long conversations can add latency: in the Oulu prototype the whole chat history was sent with every request, and its cumulative size affected the model's response time.[1]

Besides recurring query costs, the 2024 survey notes that closed models are hard to replicate: "even when using the same model name", the same request can produce very different content at different times, which may lead developers to consider self-hosted models with open weights.[12]

Safety, moderation and disclosure

The 2024 survey identifies two main concerns when players interact with an LLM: toxic behavior, which a model may learn from its training corpus, and stereotypes or incorrect notions. It argues that an LLM playing an NPC should align with the game's themes and avoid any kind of toxic language or racial slurs, and reports that Ubisoft's NEO NPCs demo used the prompt defining each character to minimize toxicity and social biases.[12] Toolkits expose controls for this: NVIDIA's 2023 ACE announcement said the language models could be "protected against counterproductive or unsafe conversations via NeMo Guardrails",[13] and Horizon Worlds' Character Builder includes guardrails among each character's traits.[5]

A case outside VR drew attention to these risks in May 2025, when an AI-driven Darth Vader in Fortnite, published by Epic Games, answered players using Google Gemini 2.0 Flash and ElevenLabs Flash v2.5, with a voice imitating James Earl Jones's performance with permission from his estate. Game Developer reported that players got the character to give offensive responses, including slurs, and, crediting Kotaku, that Epic "quickly deployed a hotfix".[9]

In January 2024 Valve added an AI disclosure section to the Content Survey that developers fill out when submitting a game to Steam. The section separates AI use into two broad categories: Pre-Generated content, created with the help of AI tools during development, and Live-Generated content, created with the help of AI tools while the game is running. For Live-Generated content, developers must also tell Valve what guardrails they are putting on the AI to ensure it does not generate illegal content. Valve said it would include much of the disclosure on the game's Steam store page, released a system for players to report illegal content in games with Live-Generated AI content through the in-game overlay, and said it was unable, at the time, to release Adult Only sexual content created with Live-Generated AI.[10] A revised version of the form, reported by GamingOnLinux in January 2026, states that "efficiency gains through the use of these tools is not the focus" of the section, which is concerned with AI used to create content that ships with the game and is consumed by players; it keeps separate pre-generated and live-generated categories.[43]

The authors of the generative agents paper proposed that such agents "should explicitly disclose their nature as computational entities" and warned of people forming parasocial relationships with them.[17] Where processing happens also matters for privacy in VR and AR. Convai's guide for Meta Quest mixed reality apps states that its Vision Publisher "sends a live feed from the Quest camera to the Convai server" (see Passthrough Camera API).[44] The 2024 survey notes that conversations with models hosted via APIs are exchanged over third-party websites, while querying local models keeps all messages within the application.[12]

Labor and performer rights

SAG-AFTRA's Interactive Media Agreement covers AI-generated performances in video games. Members approved the agreement on 9 July 2025 by 95.04% to 4.96%; GamesBeat reported that it includes consent and disclosure requirements for AI digital replica use and lets performers suspend consent for the generation of new material during a strike.[11] According to a January 2026 summary by the law firm Frankfurt Kurnit Klein & Selz, the agreement runs from 8 November 2022 to 31 October 2028. Employers must give a "reasonably specific description" of a digital replica's use, including whether it will be used for "Real-Time Generation" (the article's example is an AI chatbot) and whether the performer will be asked to deal with sensitive content such as profanity or racial slurs. Consent must be in "clear and conspicuous" writing and, for most uses, cannot be obtained at the time of initial employment for future games.[45] TheWrap's June 2025 report on the tentative agreement said that minimum rates would be set for digital replicas created from union-covered performances, and that when the performances are created for "real-time generation", such as "creating a digital replica-voiced chatbot in a video game", the performer is entitled to at least 7.5 times that minimum.[46]

In May 2025 SAG-AFTRA filed an unfair labor practice charge with the National Labor Relations Board against Llama Productions, an Epic Games subsidiary, alleging that it had replaced actors' work by using AI to generate Darth Vader's voice in Fortnite "without providing notice to the union or the opportunity to bargain".[47][9] Narrative director Virginie Mosser said of Ubisoft's Teammates experiment that she initially had "the same concerns as many others" about AI and that "I still write the story and character personalities"; instead of fixed lines, the team creates "fences" that let the characters improvise within the lore and motivations the writers have given them. Ubisoft wrote that the team was aware of criticism of AI in games and that its goal was not to replace creatives.[24]

See also

References

  1. ↑ 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 Mikko Korkiakoski, Saeid Sheikhi, Jesper Nyman, Jussi Saariniemi, Kalle Tapio, Panos Kostakos (2025-07-14). "An Empirical Evaluation of AI-Powered Non-Player Characters' Perceived Realism and Performance in Virtual Reality Environments". arXiv preprint. https://arxiv.org/abs/2507.10469. Retrieved 2026-11-01.
  2. ↑ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 Frederik Roland Christiansen, Linus Nørgaard Hollensberg, Niko Bach Jensen, Kristian Julsgaard, Kristian Nyborg Jespersen, Ivan Nikolov (2024-10-09). "Exploring Presence in Interactions with LLM-Driven NPCs: A Comparative Study of Speech Recognition and Dialogue Options". Proceedings of the 30th ACM Symposium on Virtual Reality Software and Technology (VRST '24), pp. 1-11. https://doi.org/10.1145/3641825.3687716. Retrieved 2026-11-01.
  3. ↑ 3.0 3.1 3.2 3.3 3.4 3.5 Mykola Maslych, Mohammadreza Katebi, Christopher Lee, Yahya Hmaiti, Amirpouya Ghasemaghaei, Christian Pumarada, Janneese Palmer, Esteban Segarra Martinez, Marco Emporio, Warren Snipes, Ryan P. McMahan, Joseph J. LaViola Jr. (2025-07-07). "Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual Agents". Proceedings of the 7th ACM Conference on Conversational User Interfaces (CUI '25), pp. 1-15. https://doi.org/10.1145/3719160.3736636. Retrieved 2026-11-01.
  4. ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 Nima Zargham, Leandro Tonini, Dmitry Alexandrovsky, Emma Grace Ruthven, Maximilian A. Friehs, Leon Tristan Dratzidis, Bastian Dänekas, Ioannis Bikas, Lennart E. Nacke, Sven Zebel, Rainer Malaka (2026-02-02). "Dialogs with GenAI NPCs: Exploring Player Interactions with Speech Agents in a VR Game". International Journal of Human-Computer Interaction, vol. 42, no. 19, pp. 16127-16155. https://doi.org/10.1080/10447318.2026.2620647. Retrieved 2026-11-01.
  5. ↑ 5.00 5.01 5.02 5.03 5.04 5.05 5.06 5.07 5.08 5.09 David Heaney (2025-09-10). "Horizon Worlds Creators Can Now Add Scriptable Conversational AI NPCs". UploadVR. https://www.uploadvr.com/meta-horizon-worlds-conversational-ai-npcs/. Retrieved 2026-11-01.
  6. ↑ 6.0 6.1 6.2 Mike Johnson (2026-01-14). "Stellar Cafe Review: And I, For One, Welcome Our AI Overlords". UploadVR. https://www.uploadvr.com/stellar-cafe-review/. Retrieved 2026-11-01.
  7. ↑ 7.0 7.1 Don Hopper (2025-03-12). "Playing Skyrim VR With ChatGPT Powering NPC Conversations". UploadVR. https://www.uploadvr.com/playing-skyrim-vr-with-chatgpt-powering-npc-conversations/. Retrieved 2026-11-01.
  8. ↑ 8.0 8.1 8.2 8.3 Meta (2024-05-22). "Niantic Uses Meta Llama to Bring Digital Creatures to Life". Meta AI blog. https://ai.meta.com/blog/niantic-peridot-llama/. Retrieved 2026-11-01.
  9. ↑ 9.0 9.1 9.2 9.3 Chris Kerr (2025-05-20). "Voice actor union strikes back after Fortnite debuts controversial AI-powered Darth Vader". Game Developer. https://www.gamedeveloper.com/business/voice-actor-union-hits-back-after-fortnite-debuts-controversial-ai-powered-darth-vader. Retrieved 2026-11-01.
  10. ↑ 10.0 10.1 10.2 Valve (2024-01-09). "AI Content on Steam". Steamworks Development, Steam News. https://store.steampowered.com/news/group/4145017/view/3862463747997849618. Retrieved 2026-11-01.
  11. ↑ 11.0 11.1 11.2 Dean Takahashi (2025-07-09). "SAG-AFTRA union members approve end of strike with game firms over AI". GamesBeat. https://gamesbeat.com/sag-aftra-union-members-approve-to-end-strike-with-game-firms-over-ai/. Retrieved 2026-11-01.
  12. ↑ 12.0 12.1 12.2 12.3 12.4 12.5 12.6 12.7 12.8 Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, Georgios N. Yannakakis (2024). "Large Language Models and Games: A Survey and Roadmap". IEEE Transactions on Games. https://doi.org/10.1109/TG.2024.3461510. Retrieved 2026-11-01.
  13. ↑ 13.0 13.1 13.2 13.3 13.4 13.5 13.6 NVIDIA (2023-05-28). "NVIDIA ACE for Games Sparks Life Into Virtual Characters With Generative AI". NVIDIA Newsroom. https://nvidianews.nvidia.com/news/nvidia-ace-for-games-sparks-life-into-virtual-characters-with-generative-ai. Retrieved 2026-11-01.
  14. ↑ 14.0 14.1 14.2 14.3 14.4 NVIDIA (2024-06-02). "NVIDIA Releases Digital Human Microservices, Paving Way for Future of Generative AI Avatars". NVIDIA Newsroom. https://nvidianews.nvidia.com/news/digital-humans-ace-generative-ai-microservices. Retrieved 2026-11-01.
  15. ↑ 15.0 15.1 15.2 15.3 Inworld AI (2025-10-15). "The new AI infrastructure for scaling games, media, and characters". Inworld AI blog. https://inworld.ai/blog/new-ai-infrastructure-scaling-games-media-characters. Retrieved 2026-11-01.
  16. ↑ 16.0 16.1 16.2 Ubisoft (2024-03-19). "Ubisoft Unveils "NEO NPCs", its First Prototype For GenAI-powered Characters and Gameplay". Ubisoft press release. https://staticctf.ubisoft.com/8aefmxkxpxwl/Mw2s4KjssknqHHh1VIf8V/720296810ecc3deae51778a219732c7f/PRESS_RELEASE_GDC_UbisoftUnveilsNEONPC_190324.pdf. Retrieved 2026-11-01.
  17. ↑ 17.0 17.1 17.2 17.3 17.4 Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein (2023-10-29). "Generative Agents: Interactive Simulacra of Human Behavior". Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23), pp. 1-22. https://doi.org/10.1145/3586183.3606763. Retrieved 2026-11-01.
  18. ↑ 18.0 18.1 18.2 Andrew Burnes, Seth Schneider, Jason Paul (2025-01-06). "NVIDIA Redefines Game AI With ACE Autonomous Game Characters". NVIDIA GeForce News. https://www.nvidia.com/en-us/geforce/news/nvidia-ace-autonomous-ai-companions-pubg-naraka-bladepoint/. Retrieved 2026-11-01.
  19. ↑ 19.0 19.1 19.2 NVIDIA. "ACE for Games". NVIDIA Developer. https://developer.nvidia.com/ace-for-games. Retrieved 2026-11-01.
  20. ↑ 20.0 20.1 20.2 20.3 Meta (2026-07-08). "AI Building Blocks - Overview". Meta Horizon OS developer documentation. https://developers.meta.com/horizon/documentation/unity/unity-ai-building-blocks-overview/. Retrieved 2026-11-01.
  21. ↑ Haiyan Zhang (2023-11-06). "Xbox and Inworld AI partner to empower game creators with the potential of Generative AI". Microsoft Game Dev. https://developer.microsoft.com/en-us/games/articles/2023/11/xbox-and-inworld-ai-partnership-announcement/. Retrieved 2026-11-01.
  22. ↑ NVIDIA (2024-03-18). "NVIDIA Digital Human Technologies Bring AI Characters to Life". NVIDIA Newsroom. https://nvidianews.nvidia.com/news/nvidia-digital-human-technologies-bring-ai-characters-to-life-6900750. Retrieved 2026-11-01.
  23. ↑ 23.0 23.1 23.2 Dean Takahashi (2025-08-13). "Inworld Runtime is 1st AI runtime for consumer apps". GamesBeat. https://gamesbeat.com/inworld-runtime-is-1st-ai-runtime-for-consumer-apps/. Retrieved 2026-11-01.
  24. ↑ 24.0 24.1 24.2 Daniel O'Connor (2025-11-21). "Ubisoft Reveals Teammates - An AI Experiment to Change the Game". Ubisoft News. https://news.ubisoft.com/en-us/article/3mWlITIuWuu0MoVuR6o8ps. Retrieved 2026-11-01.
  25. ↑ Inworld AI. "Build with Inworld". Inworld documentation. https://docs.inworld.ai/introduction. Retrieved 2026-11-01.
  26. ↑ 26.0 26.1 Inworld AI. "Character Studio deprecation". Inworld documentation. https://docs.inworld.ai/character-studio-deprecation. Retrieved 2026-11-01.
  27. ↑ 27.0 27.1 27.2 Convai. "XR headsets". Convai documentation. https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/xr-headsets. Retrieved 2026-11-01.
  28. ↑ 28.0 28.1 Meta (2025-08-27). "Meet the Newest GenAI Tools: Environment Generation and Embodied LLM NPCs". Meta Horizon Developers Blog. https://developers.meta.com/horizon/blog/environment-generation-embodied-conversational-llm-npcs-genai-tools/. Retrieved 2026-11-01.
  29. ↑ 29.0 29.1 David Heaney (2026-03-19). "Meta Backtracks, Will Keep Horizon Worlds VR Support "For Existing Games"". UploadVR. https://www.uploadvr.com/meta-will-keep-horizon-worlds-vr-support-for-existing-games/. Retrieved 2026-11-01.
  30. ↑ 30.0 30.1 30.2 30.3 Kadir Burak Buldu, Süleyman Özdel, Ka Hei Carrie Lau, Mengdi Wang, Daniel Saad, Sofie Schönborn, Auxane Boch, Enkelejda Kasneci, Efe Bozkir (2025-01-27). "CUIfy the XR: An Open-Source Package to Embed LLM-Powered Conversational Agents in XR". 2025 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), pp. 192-197. https://doi.org/10.1109/AIxVR63409.2025.00037. Retrieved 2026-11-01.
  31. ↑ 31.0 31.1 31.2 Art from the Machine. "art-from-the-machine/Mantella". GitHub. https://github.com/art-from-the-machine/Mantella. Retrieved 2026-11-01.
  32. ↑ 32.0 32.1 Art from the Machine. "Mantella". Mantella documentation. https://art-from-the-machine.github.io/Mantella/. Retrieved 2026-11-01.
  33. ↑ Scott Hayden (2025-08-28). "'Horizon Worlds' is Getting AI-powered NPCs Soon That Talk, Guide and React in Real Time". Road to VR. https://roadtovr.com/meta-horizon-worlds-ai-npcs-update-llama-4/. Retrieved 2026-11-01.
  34. ↑ Meta (2025-09-18). "Meta Connect 2025: How AI Is Supercharging the Metaverse, Expanding Use Cases, & Our Wearable Device Access Toolkit". Meta blog. https://www.meta.com/blog/connect-2025-day-2-keynote-recap-vr-development-use-cases-wearable-device-access-toolkit/. Retrieved 2026-11-01.
  35. ↑ AstroBeam (2025-08-12). "AstroBeam Reveals Stellar Cafe, the First VR Game Powered by Real-Time AI Conversations". Triple Point PR. https://pressreleases.triplepointpr.com/2025/08/12/astrobeam-reveals-stellar-cafe-the-first-vr-game-powered-by-real-time-ai-conversations/. Retrieved 2026-11-01.
  36. ↑ Scott Hayden (2025-08-12). "'Stellar Cafe' Hands-on - 'Job Simulator' Vibes & AI Powered NPCs That Don't Actually Suck". Road to VR. https://roadtovr.com/stellar-cafe-hands-on-job-simulator-vibes-ai-powered-npcs-that-dont-actually-suck/. Retrieved 2026-11-01.
  37. ↑ Asim Ahmed (2025-10-16). "A New Sense of Direction: How Niantic Spatial's Location-Aware AI Companion is Evolving". Niantic Spatial. https://www.nianticspatial.com/blog/project-jade-lens-fest. Retrieved 2026-11-01.
  38. ↑ The Peridot team (2026-04-23). "Saying Goodbye to the Peridot Mobile Game". Peridot. Niantic Spatial. https://playperidot.com/en/news/peridot-mobile-sunset. Retrieved 2026-11-01.
  39. ↑ Liquid City Ltd.. "Wisp World". App Store. Apple. https://apps.apple.com/us/app/wisp-world/id6476198961. Retrieved 2026-11-01.
  40. ↑ "WISP World". Raindance Immersive. https://www.raindanceimmersive.com/wisp-world/. Retrieved 2026-11-01.
  41. ↑ Hongyu Wan, Jinda Zhang, Abdulaziz Arif Suria, Bingsheng Yao, Dakuo Wang, Yvonne Coady, Mirjana Prpa (2024-05-11). "Building LLM-based AI Agents in Social Virtual Reality". Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA '24), pp. 1-7. https://doi.org/10.1145/3613905.3651026. Retrieved 2026-11-01.
  42. ↑ Süeda Özkaya, Santiago Berrezueta-Guzman, Stefan Wagner (2025). "How LLMs Are Shaping the Future of Virtual Reality". IEEE Access, vol. 13, pp. 193335-193355. https://doi.org/10.1109/ACCESS.2025.3631594. Retrieved 2026-11-01.
  43. ↑ Liam Squires-Hand (2026-01-17). "Valve tweak Steam AI disclosure form for developers to clarify it's for content consumed by players". GamingOnLinux. https://www.gamingonlinux.com/2026/01/valve-tweak-steam-ai-disclosure-form-for-developers-to-clarify-its-for-content-consumed-by-players/. Retrieved 2026-11-01.
  44. ↑ Convai. "Mixed Reality AI Characters with Vision (Meta Quest)". Convai documentation. https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin-beta-overview/mixed-reality-ai-characters-with-vision-meta-quest. Retrieved 2026-11-01.
  45. ↑ Emma Smizer, Dorian Thomas (2026-01-29). "Inside the New SAG-AFTRA Interactive Media Agreement: New Standards for AI and Digital Replicas". Technology Law, Frankfurt Kurnit Klein & Selz. https://technologylaw.fkks.com/post/102mewu/inside-the-new-sag-aftra-interactive-media-agreement-new-standards-for-ai-and-di. Retrieved 2026-11-01.
  46. ↑ Jeremy Fuster (2025-06-12). "SAG-AFTRA Video Game Deal Includes AI Consent Guardrails, Minimum Rates for Digital Replica Use". TheWrap. https://www.thewrap.com/sag-aftra-video-game-deal-details/. Retrieved 2026-11-01.
  47. ↑ Sarah Parvini (2025-05-19). "SAG-AFTRA files unfair labor practice charge over use of AI to make Darth Vader's voice in Fortnite". Associated Press, via ClickOnDetroit. https://www.clickondetroit.com/tech/2025/05/19/sag-aftra-files-unfair-labor-practice-charge-over-use-of-ai-to-make-darth-vaders-voice-in-fortnite/. Retrieved 2026-11-01.
Retrieved from "https://vrarwiki.com/index.php?title=Generative_AI_non-player_characters&oldid=53288"