How to develop VR games with AI

How to develop VR games with AI is a guide to the documented uses of artificial intelligence and machine learning in virtual reality game development. It covers AI assistants and coding agents in game engines, generated art, audio and animation, conversational characters, models that run on the headset during play, automated testing, performance budgets, and the store and legal rules that apply to AI-generated content.

The page is organized by development task. Each section summarizes documented tools and practices with their status and dates, and links the detailed article on the topic.

Several of these tools were renamed or changed status during 2025 and 2026, and many are offered only as betas, previews or experimental features. Unity, for example, is retiring the "Unity AI" brand name for its AI tools, which it still offers as beta versions, and Epic Games labels the plugin that connects AI agents to Unreal Engine 5.8 as experimental. Status information on this page is as of November 2026.

Reviewed 1 November 2026. All cited sources re-opened; tool status, versions, release dates, quotes, store policies, legal rulings and study figures checked against them. About review dates.

Terminology

In game development the term "AI" covers several techniques. In their textbook Artificial Intelligence and Games (Springer, 2018), Georgios N. Yannakakis and Julian Togelius describe finite state machines, behavior trees and utility-based AI as "ad-hoc behavior authoring methods that have traditionally dominated the control of non-player characters in games", and write that in the game development scene the term game AI "is still nowadays synonymous with the use of these methods". They add that industry-focused game AI textbooks tend to concentrate "almost exclusively" on what the authors call playing for experience, in particular lifelike non-player character behavior, and they name "game-playing, content generation and player modeling" as the major application areas of AI methods within games.

A survey of large language models (LLMs) in games by Roberto Gallotta and co-authors separates roles an LLM can take within a game (player, non-player character, player assistant, game master or game mechanic) from roles outside the game's runtime (automated designer, design assistant or analyst of gameplay data); an LLM can also act as a commentator on a play session or a reteller of past game events. This guide uses "machine learning at runtime" for trained models that run during play, such as object detection on headset camera images, and "generative AI tools" for models that produce code, text, speech, images or 3D assets.

AI tools in game engines

The table lists documented AI features of engines and creation tools used for VR content, with the status each vendor gives.

Engine or platform	Tool	What it does	Status and date
Unity	In-editor assistant and generators (Unity AI)	Project-aware assistant that answers questions, generates code, renames assets in bulk and places objects in a scene; generators for sprites, textures, sound, animation and materials, and from January 2026 skybox cubemaps and 3D models	Beta. Included in Unity 6.2 (August 2025) as the Unity AI Beta; open beta for Unity 6 and later from May 2026, with Ask, Plan and Agent modes.
Unity	Unity CLI and MCP server	Lets external coding agents drive the Editor; Unity also publishes an official plugin for Claude Code, Codex and Grok	Unity has deprecated the MCP server component of its assistant package; the CLI's unity mcp command replaces it.
Unity	Unity Sentis	Runtime library that runs trained neural networks inside a build	Version 2.6.1 (2 April 2026)
Unity	Unity ML-Agents Toolkit	Uses games and simulations as environments for training agents, with reinforcement learning, imitation learning, neuroevolution or other methods	Version 4.2.0 (30 October 2026)
Unreal Engine	AI assistant	Editor panel that answers questions, generates C++ code and gives step-by-step guidance; pressing F1 over an interface element starts a conversation about it	Introduced in Unreal Engine 5.7 (12 November 2025)
Unreal Engine	MCP Server plugin	Lets agentic AI systems connect to the Unreal Editor to understand the engine and the project and to help build assets or systems	Experimental, Unreal Engine 5.8 (17 June 2026)
Unreal Engine	Neural Network Engine (NNE)	Common API for running neural network models in real time, with CPU, GPU and render dependency graph (RDG) interfaces	Beta; in Unreal Engine 5.8 the NNERuntimeORT plugin moved to ONNX Runtime 1.24.3
Godot	Contribution policy	Bars substantial AI-generated code, autonomous agents and "vibe coding" in contributions to the engine, and requires contributors to disclose AI use in pull requests	Announced 30 June 2026
Blender	MCP Server (Blender Lab add-on)	Natural-language interface to Blender's Python API; examples on its page include analyzing a scene for performance problems and suggesting names for data-blocks	Add-on version 1.0.3, requires Blender 5.1 or newer
Horizon Worlds	Worlds Desktop Editor generative AI tools	Generates meshes, textures, skyboxes, sound effects, ambient audio and TypeScript; Environment Generation creates "an entire themed environment" from a text prompt	Environment Generation available to all creators (August 2025); UploadVR reported the generation features as available in the US, UK, Canada, EU, Australia and New Zealand
Meta Horizon	Horizon Create and Horizon Studio	A mobile app and a browser-based tool that turn text prompts into complete 2D or 3D mobile games on the Meta Horizon Engine	In testing with creators; early access sign-ups from September 2026, rolling out gradually, with no launch date given
Meta XR SDK for Unity	Meta XR AI Building Blocks	Plug-and-play agents for LLM chat, speech to text, text to speech and object detection; inference runs in the cloud, on a local machine or on the headset, depending on the provider	Added in v83 (December 2025)
Meta XR SDK	Meta XR Operator	OpenXR API layer that exposes a running app's state and input to an AI agent over MCP	Experimental component of Meta XR SDK v205, announced 18 August 2026
Meta	Immersive Web SDK	Open-source web framework built on Three.js; every new project includes an MCP server and agent skills, and its Agentic Scene Composer is a visual editor that an agent can drive	Version 1.0.0, the first stable release, published on npm on 24 September 2026 and announced on 14 October 2026
Snap Lens Studio	Agentic development for Specs Lenses; Migration Agent	Brings Lens Studio agents and skills into AI tools such as Claude Code, Codex and Cursor; Migration Agent ports projects from tools such as Unity into Lens Studio	Agentic development in developer preview (16 June 2026)

Meta's creation tools for Horizon Worlds changed direction during this period. The Worlds Desktop Editor is a Windows application that Meta released in early access in February 2025, when it deprecated the in-VR creation tools of Horizon Worlds; in September 2025 UploadVR reported that Meta would rebrand the editor as Horizon Studio, with a beta "in the coming months". In February 2026 Meta said it was "explicitly separating our Quest VR platform from our Worlds platform" and "doubling down on the VR developer ecosystem while shifting the focus of Worlds to be almost exclusively mobile". At Meta Connect 2026 it presented Horizon Create, a standalone mobile app, and the browser-based Horizon Studio as tools that let creators "turn any idea into a complete 2D or 3D mobile game", with distribution across Facebook, Instagram and Horizon.

AI-assisted programming

Vendors document two forms of AI help with engine programming: assistants built into the editor, as in Unity and Unreal Engine, and external AI coding agents that control the editor or the headset, often through the Model Context Protocol (MCP). Anthropic introduced MCP on 25 November 2024 as an open standard for two-way connections between AI tools and data sources.

Unity's command-line interface "drives a local Editor offline", and its local pipeline server "accepts connections on localhost only"; Unity states that its direct unity command and unity eval commands are "faster and use fewer tokens than MCP". Blender has no built-in functionality for connecting to LLMs: the Blender Lab MCP Server needs an add-on, an LLM client and the server itself, and its page warns that the server "will execute LLM generated code in Blender without any guards in place" and recommends "a virtual machine, or a system without access to sensitive information".

For Horizon OS development, Meta's agentic-tools repository (Apache 2.0 license) packages agent skills around the metavr command-line tool. The tool includes an MCP server with "40+ tools", and the skills are packaged for Claude Code, Cursor, Codex, GitHub Copilot CLI and Gemini-compatible environments. Meta says the tools in Immersive Web SDK 1.0 let an agent "see, drive, and debug" a WebXR app, and that the release already supports building for the Meta VR Glasses before they ship in spring 2027. For its developer preview of agentic development in Lens Studio, Snap says developers "stay in control by reviewing suggestions, steering the creative direction, and making final decisions".

Projects also set rules for AI-written code. When it announced new contribution rules for the Godot engine in June 2026, the Godot Foundation wrote "We require all code to be human authored", limited AI assistance to "menial things (like code completion, regex, or find and replace)" and said that "autonomous AI agent use or vibe coding" already leads to an automatic ban from the project's GitHub repository. After Anthropic joined the Blender Development Fund as a corporate patron and community members raised concerns about how the project engages with AI companies, the Blender Foundation said on 1 May 2026 that it would receive the funds "as a singular donation instead of a Development Fund membership", that it needed to define how Blender is positioned on generative AI, and that "No generative AI functionality is currently available or planned to be integrated in Blender."

Published productivity measurements come from general software development tasks. A randomized controlled trial by METR (arXiv preprint, July 2025), in which 16 experienced developers completed 246 tasks in mature open-source projects, found that allowing early-2025 AI tools, mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet, increased completion time by 19%; the developers had forecast a 24% reduction and afterwards estimated a 20% reduction. Reporting on a follow-up study with 57 developers and more than 800 tasks on 24 February 2026, METR wrote that developers were likely sped up more in early 2026 than in early 2025, but that because of selection effects its data was "only very weak evidence for the size of this increase": more developers declined to take part because they did not want to work without AI. METR changed its study design in response.

Generating 3D assets, textures and environments

Main article: Text-to-3D generation

In Unity 6.2 the Generators toolset combined sprite, texture, sound, animation and material generation; the January 2026 beta added a skybox generator that creates cubemaps from a prompt or an image and a 3D Model Generator that works from text or a reference image. Among the Snap OS updates it announced in June 2025, Snap listed a Snap3D API that "lets developers generate 3D objects on the fly inside Lenses".

Skybox AI covers a dedicated tool for generating 360-degree skyboxes, and Gaussian splatting covers photoreal scenes reconstructed from photographs.

Characters, dialogue and voice

Main article: Generative AI non-player characters

Conversational characters in VR combine speech recognition, an LLM and speech synthesis. In a between-subjects study presented at the ACM Symposium on Virtual Reality Software and Technology (VRST) in 2024, Frederik Roland Christiansen and co-authors had participants play a 15-minute murder mystery. The game ran on a laptop with an Oculus Quest 2 as the headset; in both conditions the characters' replies came from GPT-4 (gpt-4-1106-preview), voiced with the Oculus Voice SDK. One group spoke to the characters, with speech transcribed by the Whisper API; the other chose dialogue options. Data from 22 of the 24 participants (11 per condition) were analyzed. Measured over 20 test questions per condition, the average response time was 4.00 seconds with speech recognition, mainly because of the transcription step, and 2.66 seconds with dialogue options. Both versions gave similar social presence; users felt more immersed with speech recognition but found it overwhelming, and slow NPC responses were a source of frustration.

Platform SDKs supply the pieces. Meta's Voice SDK for Unity, powered by the Wit.ai natural language understanding service, provides automatic speech recognition, which turns voice requests into text, and natural language processing, which turns text into user intents. The Meta XR AI Building Blocks add agents for LLM chat, speech to text and text to speech. In September 2025 Meta made conversational NPCs powered by its Llama 4 model available to all Horizon Worlds creators, but only adult users in the United States could talk to them; for other users, creators could implement a fallback that speaks scripted lines through text-to-speech and bypasses the LLM.

NVIDIA ACE, as NVIDIA described it in March 2024, combines Riva automatic speech recognition and text-to-speech with Audio2Face facial animation, and NVIDIA says its microservices can run "across cloud and PC depending on the local GPU capabilities". Photoreal characters of this kind are covered in Digital human, and lip sync generated from speech in Audio-driven facial animation.

Players also add conversational NPCs to existing games through mods. Mantella, an open-source (AGPL-3.0) mod for Skyrim and Fallout 4, chains speech-to-text, an LLM and text-to-speech. An UploadVR writer who tested Mantella in Skyrim VR in March 2025 reported "significant lag time" in spoken responses with the default free OpenRouter account, much shorter waits after switching to a paid API key with the GPT-4o mini model, and NPCs that sometimes failed to respond, especially when several AI-enhanced characters were addressed at once in a crowd.

Animation and avatars

Main article: Markerless motion capture

For offline animation, Unreal Engine 5.8 added an experimental MetaHuman Animator capability that captures the body, or face and body, performance of a single actor from a single camera, with an offline solve, on consumer hardware "including webcams and supported smartphones".

For real-time avatars, Meta's Movement SDK estimates parts of the body that the headset does not track. Meta says Generative Legs, released in December 2023 for all Meta Quest devices, produces "plausible leg motions" with the help of AI by assessing the position of the upper body; it was introduced together with Inside-Out Body Tracking, which Meta then described as exclusively available on the Meta Quest 3. The current documentation also describes an AI Motion Synthesizer that "produces natural character motion through a sparse set of signals from the Quest headset". Research methods for this problem are covered in Full-body pose estimation from sparse tracking; related topics are Body tracking, Full-body tracking, Motion capture and Inverse kinematics.

Procedural content and difficulty

Main article: Procedural generation

Main article: Dynamic difficulty adjustment

Epic Games declared the Procedural Content Generation (PCG) framework production-ready in Unreal Engine 5.7, describing it as a way to populate environments quickly and introduce natural variety. Meta says mobile games made with Horizon Create and Horizon Studio come "complete with progression systems, balanced difficulty, art direction, multiplayer, and more"; when it announced the tools in September 2026 they were being tested with creators, and early access was rolling out gradually. For LLMs in VR games, a 2025 scoping review in IEEE Access names procedurally generated storytelling and AI-driven adaptive systems among the "major application domains" it outlines.

On-device machine learning and mixed reality perception

Engines include inference libraries for running trained models inside a build. Sentis, Unity's inference library, supports most models in the ONNX (opset 7 to 25) and LiteRT formats, as well as PyTorch models that meet its operator requirements. It runs them in real time on the target device's CPU or GPU, and Unity says it supports real-time applications on all Unity-supported platforms. Behaviors trained with ML-Agents are embedded in a scene through Sentis.

On Meta Quest, the Passthrough Camera API gives apps images from the forward-facing RGB cameras for computer vision and machine learning. It requires a Meta Quest 3 or Meta Quest 3S with Horizon OS v74 or later and a camera permission. The data rate is 60 Hz and image capture latency is 20-40 ms; the maximum resolution, 1728 x 1728 pixels, was added with version 209, and Meta advises apps not to rely on a particular resolution. Meta's Unity samples for the API include a MultiObjectDetection scene that recognizes real-world objects with Unity's inference library, and they state that the Meta XR Simulator does not currently support the camera API. Meta's recommended setup for the Meta XR AI Building Blocks is Unity 6 or newer, a Quest 3 or 3S, and version 83 or later of the Meta XR Core SDK and, for passthrough camera support, the MR Utility Kit; each agent uses a provider in the cloud, on a local machine or on the headset, and from version 85 the building blocks can run text generation models such as SmolLM, Qwen and Phi directly on Quest. Related topics are Passthrough, Computer vision, Scene understanding and mixed reality.

Apple's Foundation Models framework gives apps access to the on-device large language model at the core of Apple Intelligence; when announcing it in June 2025, Apple described this inference as "free of cost" and available when users are offline. The framework's documentation lists visionOS 26 among the supported platforms, says on-device models "excel" at text generation tasks that include "dialog for games", and points to Private Cloud Compute or any server model provider when more reasoning capability or context size is needed. It includes sample code to "make gameplay more lively with AI generated dialog and encounters personalized to the player", and states that people need a device that supports Apple Intelligence. For Android XR, Google documents access to the Gemini APIs through Firebase AI Logic for native Kotlin apps and for Unity. These call cloud-based Gemini models; Google suggests the Gemini Live API with function calling for voice navigation, and structured output for game interactions in Unity apps. In June 2025, when it said it would launch Specs in 2026, Snap announced Snap OS updates for Lens developers: integrations with OpenAI and Gemini on Google Cloud for multimodal AI-powered Lenses, an Automated Speech Recognition API supporting more than 40 languages, and a Depth Module API that "translates 2D information from large language models in order to anchor AR information accurately in three dimensions".

Some headset chips include a neural processing unit. For the M5 version of the Apple Vision Pro, Apple says the 16-core Neural Engine runs AI-powered features in third-party apps up to twice as fast as the previous generation, based on its September 2025 tests of preproduction units with text-to-image generation in the Draw Things app. Cloud inference has its own documented requirements: Meta recommends a "stable internet connection" for its building blocks when they run cloud inference or local inference through HTTP ports, and warns that, especially for cloud providers, "models may not always be available on the provider's servers". Rules on camera and sensor data are discussed in Privacy in virtual and augmented reality.

Testing and quality assurance

Main article: Automated testing of VR applications

Platform holders have released tools that let AI agents operate a VR app during testing. Meta XR Operator, an experimental OpenXR API layer in the Meta XR SDK (v205), sits between the app and the OpenXR runtime and exposes state and input over MCP. It is integrated into the Meta XR Core SDK for Unity, and developers using other OpenXR-based engines can integrate a standalone package; Meta recommends running it with Meta XR Simulator and says it can also be used on a Meta Quest with additional setup. An agent can take screenshots of the app as rendered in Meta XR Simulator, press buttons, move thumbsticks, set the head pose, and read room geometry and the Unity scene hierarchy. Meta lists its limits: the agent uses "a single aim-ray per hand with no per-finger data", has no access to audio, works from screenshots and cannot watch animations play out, and operates too slowly for real-time interaction, so it "works best with static, deterministic scenarios". The skills and metavr tool in Meta's agentic-tools repository capture screenshots and screen recordings, read device logs and profile apps with Perfetto traces and simpleperf; metavr's vrc-local command group includes store-listing and performance checks. With Immersive Web SDK 1.0, an agent can run a web project on a Meta Quest plugged into the computer and "test it with the same commands".

Trained agents are a second route: Unity's ML-Agents documentation lists "automated testing of game builds" among the uses of trained agents, alongside controlling NPC behavior and evaluating game design decisions before release.

Performance and comfort constraints

Store requirements set minimum frame rates. Meta's VRC.Quest.Performance.1 requires apps to maintain a rendering rate of at least 60 frames per second and interactive apps to use a refresh rate of 72, 80, 90, 96, 100 or 120 Hz; interactive apps may render at half the refresh rate in parts that use Application SpaceWarp. Related comfort issues are covered in Simulator sickness.

Model inference has to fit inside that frame budget. Sentis documentation gives the example of a model that takes 50 milliseconds: run in one frame it "might cause stuttering or low frame rates in gameplay", so the documentation shows how to run the model one layer at a time and spread it over 10 frames at 5 milliseconds each. The object detection agent in Meta's building blocks has a configurable detection frequency, detectEveryNFrames, where 1 runs detection every frame and 2 every other frame. Meta lists the cost of the Passthrough Camera API itself as about 1-2% GPU overhead per streamed camera and about 45 MB of memory.

Conversation latency is a separate constraint: the authors of the VRST 2024 study point to the "need for faster generation or better masking for a seamless experience".

Store policies, disclosure and rights

Store rules

In January 2024 Valve added an AI disclosure section to the content survey that developers complete when submitting a game to Steam. It separates "Pre-Generated" content, created with the help of AI tools during development, which Valve said it would evaluate the same way as all non-AI content, from "Live-Generated" content, created while the game is running, for which developers must describe the guardrails that keep the AI from generating illegal content. Valve said it would include much of the disclosure on the game's store page, let players report illegal live-generated content through the in-game overlay, and said it was unable at that time to release Adult Only sexual content created with live-generated AI. In January 2026 Valve revised the form to say that "Efficiency gains through the use of these tools is not the focus of this section" and that the section concerns AI used to create content that ships with the game and is consumed by players, such as artwork, sound, narrative and localization.

The content guidelines of the Meta Horizon Store (last updated 27 June 2025) do not mention AI specifically; they apply to all content a developer generates, recommends or submits as part of an app. Apple's App Review Guidelines (last updated 16 October 2026) list "AI content creation apps" under guideline 1.2 on user-generated content, which requires filtering of objectionable material, a way to report offensive content, the ability to block abusive users and published contact information. Guideline 5.1.2(i) requires apps to disclose clearly where personal data will be shared with third parties, "including with third-party AI", and to obtain explicit permission first. Google Play's AI-Generated Content policy requires apps that generate content using AI to prohibit and prevent the generation of restricted content and to include in-app reporting or flagging features, so that users can report offensive content to developers without leaving the app. Unity leaves such declarations to developers: it says assets generated by its in-editor assistant "contain embedded metadata flagging them as AI-generated" and that developers "are fully responsible for managing app store declarations and verifying usage rights".

Copyright and performers

Part 2 of the US Copyright Office report Copyright and Artificial Intelligence (29 January 2025), on copyrightability, concludes that copyright does not extend to purely AI-generated material, that "prompts alone do not provide sufficient human control" given current generally available technology, and that human authors are entitled to copyright in their own perceptible expression and in their creative selection, coordination, arrangement or modification of AI-generated material, assessed case by case. In Thaler v. Perlmutter, the US Court of Appeals for the District of Columbia Circuit held on 18 March 2025 that the Copyright Act "requires all eligible work to be authored in the first instance by a human being", while stating that this requirement "does not prohibit copyrighting work that was made by or with the assistance of artificial intelligence". The Supreme Court denied review on 2 March 2026.

Members of the performers' union SAG-AFTRA ratified the 2025 Interactive Media Agreement on 9 July 2025 by 95.04% to 4.96%, ending the union's video game strike. The agreement includes "consent and disclosure requirements for AI digital replica use".

Research

In their survey of LLMs and games, Gallotta and co-authors found that the roles of player and automated designer had received the most research attention, and that the applicability of LLMs as design assistants was "somewhat limited, reverting only to conceptual assistance". The IEEE Access scoping review by Süeda Özkaya, Santiago Berrezueta-Guzman and Stefan Wagner analyzed 66 peer-reviewed studies published from 2018 to 2025 at the intersection of LLMs and VR, with a focus on VR games. It covers NPC interaction, narrative generation, intelligent game mastering, personalization and accessibility, and names real-time performance constraints, memory limitations, ethical risks and scalability barriers as challenges. In a CHI 2024 paper, Fernanda De La Torre and co-authors presented LLMR, a framework built on Unity for creating and modifying interactive mixed reality scenes from text in real time. The authors report that it "outperforms the standard GPT-4 by 4x in average error rate" on their tasks, and that participants in a usability study (N=11) had positive experiences.

Industry surveys cover games in general rather than VR. In the 2026 State of the Game Industry report, released by the organizers of the GDC Festival of Gaming (formerly the Game Developers Conference) and based on responses from more than 2,300 game industry professionals, 36% said they used generative AI tools as part of their job: 30% of respondents at game studios, compared with 58% of those at publishing companies, support teams and marketing or PR firms. Just over half (52%) said generative AI was having a negative impact on the industry, up from 30% a year earlier and 18% the year before. GDC reported research or brainstorming as the most common use (81%), followed by daily tasks and code assistance (47% each) and prototyping (35%).

See also

• Generative AI non-player characters
• Text-to-3D generation
• Skybox AI
• Markerless motion capture
• Audio-driven facial animation
• Full-body pose estimation from sparse tracking
• Procedural generation
• Dynamic difficulty adjustment
• Automated testing of VR applications
• Unity AI
• Unity Sentis
• Unity ML-Agents Toolkit
• Passthrough Camera API
• Meta XR AI Building Blocks
• Immersive Web SDK
• How to get started in VR development
• Developer Resource