Toggle search
Toggle menu
5.5K
6.1K
231
53.7K
VR & AR Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

How to develop AR apps with AI

vrarwiki.com · VR & AR Wiki
More actions

How to develop AR apps with AI is a guide to the documented uses of artificial intelligence and machine learning in building augmented reality (AR) applications. It covers phones and tablets, headsets with passthrough mixed reality, display AR glasses, camera-equipped AI glasses and AR on the web. It describes AI inside the running app, where models analyze camera images, answer spoken questions and generate content, and during development, in coding assistants, asset generators and test tools.

Organized by development task, the guide summarizes what platform owners and researchers had published as of November 2026 and links detailed pages. Topics shared with virtual reality development, such as AI coding assistants, text-to-3D generation and generative non-player characters, are covered in How to develop VR games with AI.

Reviewed 1 November 2026. Every cited platform document, store and EU AI Act text, press report and paper (open full text or abstract) was re-read against the claims, dates and quotes it supports. About review dates.

Terminology

Ronald Azuma's 1997 survey of the field defined AR as systems that combine real and virtual content, are interactive in real time and are registered in three dimensions.[1] Google's Android XR documentation separates XR headsets and wired XR glasses from "audio glasses" and "display glasses",[2] and notes that audio glasses have no display.[3] Meta's developer documentation calls its Ray-Ban Meta, Oakley Meta and Meta Ray-Ban Display models "AI glasses".[4]

The guide distinguishes three senses of "AI": classic computer vision techniques; machine-learning models that run inside the app, for example to segment or label camera images; and generative AI, including large language models (LLMs), which produces text, speech, images or 3D content.

Platforms also differ in where inference runs. Apple's Foundation Models framework works with on-device models and Apple's Private Cloud Compute,[5] and several client SDKs of Google's Firebase AI Logic offer hybrid and on-device inference, in which an app uses an on-device model when one is available and falls back to a cloud-hosted model when needed.[6]

Platforms and their AI interfaces

Platform AR framework Machine-learning and AI interfaces Developer camera access Status and date
iOS and iPadOS ARKit, RealityKit Vision framework and Core ML models applied to camera frames;[7] Core AI from version 27;[8] Foundation Models framework[5] Camera image in each ARKit frame, after the user grants camera permission[9][10] iOS 27 and iPadOS 27 release date 14 September 2026[11]
visionOS (Apple Vision Pro) ARKit, RealityKit Object tracking with reference objects trained in Create ML;[12] Core AI from version 27[8] Main camera only through an enterprise entitlement and license, for business use[13][14] Stereo main-camera access since visionOS 26[15]
Android phones and tablets ARCore ML Kit vision APIs;[16] ML Kit GenAI APIs on Gemini Nano;[17] MediaPipe;[18] Gemini models through Firebase AI Logic[6] ARCore CPU image (640x480 by default) as model input[19] ML Kit GenAI APIs labeled alpha, beta or preview (29 October 2026)[17]
Android XR headsets and glasses Jetpack XR SDK Gemini models through Firebase AI Logic, including the Gemini Live API for voice[20] Glasses camera through CameraX with a glasses-specific runtime permission;[21][22] on headsets, passthrough camera access through Camera2 and CameraX, as reported during the developer preview in February 2025[23] ARCore for Jetpack XR and Jetpack SceneCore 1.0.0; XR Projected 1.0.0-alpha14 (21 October 2026)[24][25][26]
Meta Quest 3 and Meta Quest 3S Meta XR Core SDK and MR Utility Kit for Unity[27] AI Building Blocks (object detection, LLM, speech to text, text to speech);[27] Unity Sentis[28] Forward-facing cameras through the Passthrough Camera API[29] Public release in Horizon OS v76, with Horizon Store publishing (April 2025)[28]
Meta's AI glasses: Ray-Ban Meta (Gen 1 and Gen 2), Ray-Ban Meta Optics, Meta Glasses, Oakley Meta HSTN, Oakley Meta Vanguard, Meta Ray-Ban Display[30] Meta Wearables Device Access Toolkit for iOS and Android apps; web apps on the Meta Ray-Ban Display[4] Data processed by the app locally or on cloud or edge platforms;[4] WebMCP (developer preview) to define what Meta AI can control in an app[31] Camera, microphone, audio and motion data through the toolkit[4] Toolkit 1.0 rolling out from 30 September 2026; submissions for publishing not yet open[4]
Spectacles and Specs Lens Studio, Snap OS SnapML for custom models;[32] Remote Service Gateway for OpenAI, Google and Snap-hosted models[33] Camera Module frames; a Lens that sends frames to a backend needs Transparent Permission to be publishable[34] Specs preorders opened 16 June 2026, with shipping planned for fall 2026 in the US, UK and France[35]
Web browsers WebXR and WebAR frameworks MediaPipe Tasks for the web;[18] Firebase AI Logic JavaScript SDK[6] WebXR Raw Camera Access in Chrome and Edge, not in Firefox or Safari[36] Raw Camera Access is a draft Community Group report (11 December 2025)[37]

Camera access and permissions

Apps that run their own models on images of the surroundings need camera frames, and each platform documents different rules for access.

Phones and tablets

On iOS and iPadOS, ARKit asks the user for camera permission the first time an app runs an AR session, and the app must supply a usage message in its NSCameraUsageDescription key; apps that use ARKit face tracking must have a privacy policy that describes how they use face data.[9] Each ARKit frame carries the camera image as a full-range YCbCr pixel buffer.[10] On Android, ARCore provides a CPU image for image processing, 640x480 by default, and a GPU texture, usually 1080p, for the camera preview; the SharedCamera API adds further camera streams.[19]

Headsets

On Apple Vision Pro, access to the forward-facing main camera is one of Apple's enterprise APIs. It needs the com.apple.developer.arkit.main-camera-access.allow entitlement and a license file,[14] and the app must provide an NSMainCameraUsageDescription message. Before visionOS 26, the CameraFrameProvider that delivers camera frames was available only in an immersive space.[13] The license is requested by the Account Holder of an organization's Apple Developer Program or Apple Developer Enterprise Program membership, not by developers enrolled as individuals, and the apps must be for business use and distributed privately as in-house apps or as custom apps through Apple Business Manager.[14]

Meta's Passthrough Camera API gives apps on the Quest 3 and Quest 3S access to the forward-facing cameras for computer vision and machine learning. It needs Horizon OS v74 or later and either android.permission.CAMERA or horizonos.permission.HEADSET_CAMERA, which Meta says gives access only to the passthrough camera. Meta lists a maximum resolution of 1728x1728, a data rate of 60 Hz and an image capture latency of 20-40 ms, and treats the images as Device User Data under its Developer Data Use Policy.[29]

Google's Android XR permissions page lists eye, face, hand and body tracking, scene understanding and precise location as dangerous permissions, which an app declares in its manifest and requests at runtime. The coarse scene-understanding permission covers plane, image, marker, QR code, mesh and object tracking, and the fine one covers depth textures and scene meshing.[38]

Glasses

On Android XR audio and display glasses, the camera needs a "glasses-specific" permission that an app must request at runtime even if it already holds the same permission on the phone; ProjectedActivityCompat.requestPermissions shows a permission dialog on the phone that states the permission is for the glasses.[21] Phone code obtains a projected context for the glasses' camera through CameraX, with ImageCapture for still images or VideoCapture and ImageAnalysis for streams, and microphone audio arrives over the Bluetooth Hands-Free Profile. Google's example settings range from 1920x1080 at one frame per second for text recognition (OCR) or world sensing, and 1024x768 at one frame per second for AI video streaming to multimodal assistants, to 640x360 or 640x480 at 10 frames per second for localization and visual positioning. Google advises choosing the lowest resolution and frame rate that work for the use case, to conserve battery power and reduce heat on the glasses, and querying the supported values at runtime because they vary by device.[22]

On Spectacles, Snap treats the camera, microphone and location as sensitive user data. Lens Studio declares permissions automatically when a Lens uses a capability, and when an API returns only processed data, such as face landmarks, hand joints or surfaces, the camera is not considered to be used. Snap's permissions overview says that when Lenses communicate with external devices through Connected Lenses or the RemoteServiceModule, Spectacles prevents simultaneous use of sensitive user data in published Lenses, and that some internal Snap Cloud services, such as the Remote Service Gateway, do not count as external connectivity and can be used with sensitive user data.[39] A Lens that sends camera frames to a backend, such as a machine-learning or LLM service, combines camera access with internet access and needs Transparent Permission to be publishable.[34] Transparent Permission lets published Lenses combine internet connectivity with sensitive user data: a dialog at launch lists the sensitive data the Lens will access, and if access is granted, the device's LED blinks while that data is captured. Snap supports only certain combinations of permissions, among them the Remote Service Gateway with the Camera Module, and Lenses that use Experimental APIs still cannot be published to a wider audience.[40]

Web

The WebXR Raw Camera Access Module, a draft W3C Immersive Web Community Group report dated 11 December 2025, lets an immersive AR session request camera images aligned with the pose of each view. The draft rates this as the WebXR capability with the highest privacy implications, says browsers should obtain user consent first, and requires a privacy indicator while camera access is active.[37] The web-features project lists support in Chrome, Chrome for Android and Edge from version 107 and none in Firefox or Safari; Mozilla's standards position is "defer" and WebKit's is "oppose".[36] Google's list of WebXR features in Chrome for Android XR does not include raw camera access, and Google states that all WebXR APIs there require the "3d mapping and camera tracking" permission.[41]

Perception with machine learning

Main article: Object recognition

Planes, meshes and room-level semantics are covered in Scene understanding and Plane detection.

Segmentation and depth

ARCore's Scene Semantics API uses a machine-learning model to give each pixel of an outdoor camera image a label, such as sky, building, tree, road, vehicle, water or person, with a confidence value. Google states that the API supports outdoor scenes only and should be used in portrait orientation; it runs on the same devices as ARCore's Depth API and has guides for Android, the Android NDK, iOS and Unity.[42] Per-pixel labeling of this kind is known as semantic segmentation. ML Kit includes selfie segmentation and subject segmentation.[16]

ARCore's Depth API builds depth images with a depth-from-motion algorithm that compares camera images taken from different angles as the device moves. Google says the algorithm "selectively uses machine learning to increase depth processing, even with minimal motion from a user", and that where a device has a dedicated depth sensor, such as a time-of-flight sensor, it merges data from all available sources. Google describes occlusion, drawing virtual objects behind real ones, as a use of this data.[43] In Chrome on Android XR, depth sensing through the WebXR Depth Sensing Module is stereoscopic and streams two depth maps.[41] See Depth sensing and Depth map.

Object detection and tracking

Google's ARCore machine-learning guide describes passing the CPU camera image to a model such as ML Kit's on-device object detection and tracking API. The results, in image-pixel coordinates, are converted to view coordinates with Frame.transformCoordinates2d(), hit-tested against the scene and turned into anchors, and its Kotlin sample labels real-world objects with ML Kit and the Cloud Vision API. To save processing power and energy, Google advises against running the model on every incoming frame and suggests running object detection at a low frame rate instead.[19]

Apple's "Tracking and altering images" sample uses the Vision framework to find rectangles likely to be artwork or photos. It checks the camera feed no more than 10 times a second, because Vision requests "can be taxing on the processor", and runs one request at a time; ARKit then tracks the cropped image as a reference image, and a bundled Core ML model redraws it in one of eight styles.[7]

For known rigid objects, Apple's object tracking uses a reference object trained in Create ML from a USDZ model, which can come from computer-aided design software or the Object Capture feature of the Reality Composer app; Apple asks for a model as photorealistic and as precise in scale as possible, like a digital twin of the object. Training needs a Mac with Apple silicon and macOS 15 or later; Apple says it can take a few hours, and the time varies with the Mac's configuration. Object tracking needs visionOS 2 or later on Apple Vision Pro, where it works only in an immersive space, or iOS 27 or later on iPhone.[12]

On Quest, Meta's Unity samples for the Passthrough Camera API include a MultiObjectDetection scene that "uses Unity Inference Engine for real-world object recognition" and a CameraToWorld scene that transforms 2D camera coordinates into 3D world rays.[44] Unity renamed the Inference Engine package to Sentis in version 2.4.0 (22 October 2025).[45] Meta's AI Building Blocks include an Object Detection block that detects and labels real objects in passthrough or camera textures with on-device or cloud models; a detectEveryNFrames setting controls how often detection runs.[27]

In Lens Studio, SnapML imports custom models in ONNX and TFLite formats from frameworks such as PyTorch and TensorFlow, and Lens Studio comes with SnapML templates, each with a sample model, for classification, object detection, multi-object detection, style transfer, segmentation and spoken keyword detection. Models can run every frame, on a user action or on a separate thread, and Snap notes that Lenses with SnapML may take longer to review.[32] On Spectacles, a camera texture can be used as input to an ML component.[34]

Images, text and codes

Fiducial markers and image targets have their own page. ML Kit's vision APIs include barcode scanning, text recognition, image labeling and document scanning.[16] Apple's June 2026 Foundation Models update added image analysis, in which a prompt includes an image and the model uses Vision tools such as OCRTool and BarcodeReaderTool.[46]

Hands, faces and bodies

Hand tracking, Face tracking and Body tracking have their own pages. ML Kit provides face detection, face mesh detection and pose detection.[16] MediaPipe Tasks include hand, face and pose landmark detection, gesture recognition and holistic landmark detection on Android, the web, Python and iOS. Google ended support for MediaPipe's legacy solutions as of 1 March 2023; its table lists some of them, such as Hands, Pose and Holistic, as upgraded to new solutions, and support for others, such as Objectron and Box Tracking, as ended.[18]

Localization

A visual positioning system (VPS) estimates a device's position from camera images. Google's Geospatial API matches the user's surroundings against a VPS localization model built from Street View images captured over more than 15 years, in which "deep neural networks identify and describe parts of the images" that are likely to stay recognizable; the results form a 3D point cloud.[47] Niantic Spatial describes vision-based positioning "even where GPS doesn't work", metric-scale reconstructions of real places as splats and meshes, and a way to "question a scene in plain language".[48] In June 2025 Snap announced a partnership to bring Niantic Spatial's VPS to Lens Studio and Specs.[49]

Multimodal assistants and AI glasses

Main article: Meta Wearables Device Access Toolkit

Several developer platforms document ways for apps to send camera images or microphone audio to multimodal models.

Android XR and the Gemini Live API

Google's Gemini Live API provides "low-latency, real-time voice and vision interactions" with Gemini over a stateful WebSocket connection. It accepts 16-bit PCM audio at 16 kHz, JPEG images at no more than one frame per second, and text, and it returns 24 kHz audio.[50] For Android XR audio and display glasses, Google documents use of the Live API through Firebase AI Logic for both audio input and output, with function declarations that let the app handle requests from Gemini in agentic features. Google cautions that the Live API needs a persistent internet connection, incurs cost, supports a limited number of concurrent connections per project and might not suit error conditions or other critical communication, especially on audio glasses. The Android XR documentation also has separate guides for speech recognition and text to speech.[3]

Meta's AI glasses

Meta's Wearables Device Access Toolkit lets iOS and Android apps use the camera, microphone, audio, motion and IMU capabilities of the Ray-Ban Meta, Ray-Ban Meta (Gen 2), Meta Ray-Ban Display, Oakley Meta HSTN and Oakley Meta Vanguard; availability varies by device.[4] Meta's setup documentation also lists Ray-Ban Meta Optics and Meta Glasses.[30] The glasses pair through the Meta AI app, and Meta states that developers can process data locally or on cloud or edge platforms and that submissions for publishing will open "soon".[4] Meta's Connect 2026 recap describes camera capture, voice input through automatic speech recognition and motion streaming in version 1.0; web apps for the Meta Ray-Ban Display that can read device position, location and Meta Neural Band wrist motion; and WebMCP, in developer preview, which lets developers "define what Meta AI can control in your app".[31]

Spectacles

Snap's Remote Service Gateway gives Spectacles Lenses access to OpenAI's Chat Completions, image generation, image editing, text-to-speech and Realtime APIs; to Google's Gemini models, including Gemini Live, and its Imagen and Lyria models; and to Snap-hosted DeepSeek R1 chat and Snap3D text-to-3D generation. It requires Lens Studio 5.10.1 and Spectacles OS 5.062 or later, and its API tokens are tied to the developer's Snapchat account and do not expire; Snap says they must not be included in shared projects or committed to version control.[33] Snap announced the OpenAI and Gemini integrations for multimodal Lenses in June 2025, together with an Automated Speech Recognition API with support for more than 40 languages, the Snap3D API for generating 3D objects inside Lenses and a Depth Module API that, according to Snap, "translates 2D information from large language models in order to anchor AR information accurately in three dimensions".[49] Captioning in XR is covered in Immersive captions.

Models on the glasses

In June 2025 Qualcomm showed prototype glasses running Meta's Llama-3.2-1B model "fully on-device" with its Snapdragon AR1+ Gen 1 chip, UploadVR reported. Qualcomm says the chip is 26% smaller, enabling slimmer glasses arms, and UploadVR wrote that its neural processing unit (NPU) "can run small language models on-device".[51] The earlier chip is covered in Qualcomm Snapdragon AR1 Gen 1.

Generative content in AR

Lens Studio 5.0, which left beta in June 2024, introduced a GenAI Suite for creating custom ML Face Effects, generating 3D assets from a text or image prompt "in minutes", generating materials and textures, and asking an AI Assistant questions.[52] Snap's current documentation says the GenAI Suite and Creator Mode are optimized for Snapchat and may have limitations on other platforms.[53] At runtime, Spectacles Lenses can call the Snap3D, image and music generation models reached through the Remote Service Gateway.[33] See Text-to-3D generation.

TikTok's Effect House has an AI Editor for training and customizing image and video models for effects: a creator picks a base model and adds a text prompt, reference images or frames, and style settings. Its video model is limited to creators with Platinum and Diamond badges; others can request access.[54]

For captured rather than generated content, photogrammetry and Gaussian splatting reconstruct real objects and places from images. Niantic Spatial describes metric-scale splats and aligned meshes of real places "from a single room to city-scale", made from ordinary imagery,[48] and Apple's Object Capture produces USDZ models of the kind used for its object tracking.[12]

AI tools for building AR experiences

Main article: Lens Studio

Coding assistants, text-to-3D tools and engine AI tools such as Unity's, described in Unity AI, are shared with VR development and covered in How to develop VR games with AI. This section covers tools that platform owners document for AR.

Lens Studio AI Developer Mode (LSAI) lets developers chat with an AI to design a Lens, debug code, learn Lens Studio features and add custom chat tools. A separate workflow, CLAD, supports Lens development with external AI tools such as Claude Code, Cursor and Codex through ls-extensions, custom AI skills that Snap says are optimized for creating Lenses for Specs.[53] In June 2026 Snap began a developer preview of agentic development that brings Lens Studio agents and skills into Claude Code, Codex and Cursor. It also released a SPECS Spatial Benchmark to show "how AI models perform on real-world spatial tasks", and said its Migration Agent "helps port projects from tools like Unity into Lens Studio".[55] UploadVR reported that Snap announced full Model Context Protocol (MCP) support for AI coding agents such as OpenAI Codex and Claude Code. It also reported that Specs Lenses are developed in Lens Studio with JavaScript or TypeScript and that the glasses cannot install APKs, so developers cannot run entirely native-code apps or use third-party engines such as Unity; the same day, Snap announced a Native Development Kit for bringing native C and C++ code and libraries into Lenses.[35]

Apple's Xcode has a coding assistant that works with agents and chat models to generate code, explore unfamiliar codebases and fix or refactor existing code.[56] Its Intelligence settings, which include a Model Context Protocol section, can turn on Claude or ChatGPT where available, add other agents that support the Agent Client Protocol, and add chat providers hosted on the internet or locally on the Mac that support the Chat Completions API.[57]

Google describes Gemini in Android Studio as an "AI-powered coding companion for Android development", and its documentation covers Agent Mode, MCP servers and the use of local or remote models.[58] Meta's Connect 2026 recap suggests pointing a coding agent at Meta's developer documentation to scaffold a glasses app, and describes skills that accompany its UI Toolkit for Meta Ray-Ban Display web apps.[31]

Meta's Immersive Web SDK (IWSDK) describes itself as "built from the ground up for AI-assisted immersive web development". Its MCP server lets coding agents such as Claude and Cursor control a running WebXR app through a managed Chromium browser driven by Playwright, which gives the agent semantic snapshots, screenshots and performance diagnostics, and through IWER, an emulated XR runtime that can move controllers and trigger input.[59]

On-device and cloud inference

Apple's Core AI framework, listed from version 27 of iOS, iPadOS and visionOS, runs AI models on the CPU, GPU and Neural Engine of Apple silicon. Models are converted to the .aimodel format with Core AI PyTorch Extensions and can be compiled ahead of time with the coreai-build tool, and a Core AI instrument profiles inference performance in Xcode.[8]

The Foundation Models framework works with Apple's on-device and Private Cloud Compute models and, through a LanguageModel protocol added in June 2026, with other server or on-device language models.[46] Apple says the on-device models "excel" at text generation tasks such as summarization, entity extraction, text and image understanding, refinement and dialog for games; the framework provides guided generation of Swift data structures and tool calling, and Apple's own models require a device that supports Apple Intelligence.[5] A Foundation Models instrument shows latency, prompts, model output, tools and token usage. In February 2026 Apple added a property that reports the on-device model's maximum context size, and Xcode's #Playground macro shows an estimate of usage of the 4,096 tokens in the available context window.[46]

Google renamed TensorFlow Lite to LiteRT in September 2024 and states that it runs models authored in PyTorch, JAX and Keras.[60] ML Kit's GenAI APIs run on Gemini Nano through AICore, an Android system service that manages on-device foundation models. As of 29 October 2026 Google's documentation labeled direct model access with custom prompts and the image description, summarization, proofreading and rewriting APIs beta; agentic workflows, speech recognition, speech translation and text to speech alpha; and embeddings preview.[17] Firebase AI Logic, which Google says was formerly called Vertex AI in Firebase,[61] provides client SDKs for Swift, Kotlin and Java, JavaScript, Dart for Flutter, and Unity, and Google says Firebase App Check helps protect the APIs that access Gemini models from abuse by unauthorized clients.[6] MediaPipe's solutions also include an LLM Inference API.[18]

Unity's Sentis (version 2.6.1) is a neural network inference library that runs models on the target device across all Unity-supported platforms. It imports most models in ONNX format with opset versions 7 to 25, most LiteRT models, and PyTorch exported programs decomposed to Core ATen IR operators.[62] On Quest, Meta's AI Building Blocks use providers that run inference in the cloud, on a local machine or on the headset; the page names OpenAI, Llama API, Hugging Face, ElevenLabs, Ollama and Unity Inference Engine, which runs "optimized models directly on Meta Quest hardware". From version 85, text generation models such as SmolLM, Qwen and Phi can run directly on Quest. Meta recommends a stable internet connection for cloud inference or local HTTP inference and warns that "models may not always be available on the provider's servers".[27]

On Apple Vision Pro, an enterprise entitlement that gives apps more CPU and GPU power comes with what Apple describes as increased thermal usage and reduced battery life.[14]

Testing and evaluation

ARCore's Recording and Playback API records an AR session to an MP4 file with the 640x480 camera image used for motion tracking, gyroscope and accelerometer data, a depth video for preview purposes and custom data tracks. ARCore can replay it in place of the live camera, so that changes can be tested without returning to the location; the API is available for Android, the Android NDK, Unity (AR Foundation) and Unreal Engine.[63] On iOS, a session recorded in the Reality Composer app can drive an ARKit app run from Xcode, through the "Replay data" scheme option, in place of the device's camera and sensors.[64]

Simulators do not cover every camera-based feature. Apple's object tracking sample must run on Apple Vision Pro because Simulator does not support object tracking,[65] and Meta's Passthrough Camera API samples note that the XR Simulator does not currently support the API.[44] Meta Ray-Ban Display web apps have a browser-based simulator,[31] and IWSDK's IWER runtime emulates XR input for agents that test WebXR apps.[59]

Models also change between releases: Apple asks developers to test prompts again when the on-device model changes with an operating system update,[46] and to retrain reference objects with the latest version of Create ML to pick up its improvements.[65] Automated testing methods are covered in Automated testing of VR applications.

Store policies, privacy and disclosure

Wider issues are covered in Privacy in virtual and augmented reality and Ethics of augmented reality.

Platform rules

Apple's App Review Guidelines, in the version last updated on 16 October 2026, state in guideline 5.1.2(i) that apps "must clearly disclose where personal data will be shared with third parties, including with third-party AI", and obtain explicit permission before doing so;[66] TechCrunch reported in November 2025 that Apple had added this sentence.[67] Guideline 5.1.2(vi) bars using data from depth and facial mapping tools such as ARKit, camera APIs and photo APIs for marketing, advertising or use-based data mining; 2.5.14 requires explicit consent and a clear visual or audible indication when an app records user activity; under 4.2.1, ARKit apps "should provide rich and integrated augmented reality experiences", and merely dropping a model into an AR view or replaying an animation is not enough; and 1.2 applies its rules for user-generated content to AI content creation apps.[66]

Google Play's AI-Generated Content policy gives text-to-text conversational chatbots and images or video generated from text, image or voice prompts as examples of AI-generated content. Apps that generate content using AI must comply with Google Play's existing developer policies, including by prohibiting and preventing the generation of restricted content, and must contain in-app reporting or flagging features that let users report offensive content to developers without needing to exit the app.[68]

For its glasses toolkit, Meta points developers to its Wearables Developer Terms and Acceptable Use Policy and limits full capabilities to countries where its AI glasses are supported.[4]

EU AI Act

Article 50 of the European Union's AI Act sets transparency obligations for providers and deployers of certain AI systems. Under Article 50(1), providers of AI systems intended to interact directly with people must ensure that those people are informed that they are interacting with an AI system, unless this is obvious to a reasonably well-informed, observant and circumspect person in the circumstances and context of use; under Article 50(2), providers of systems that generate synthetic audio, image, video or text must ensure that the outputs are "marked in a machine-readable format and detectable as artificially generated or manipulated"; and under Article 50(4), deployers of systems that generate or manipulate image, audio or video content constituting a deep fake must disclose that the content has been artificially generated or manipulated.[69] Article 113 sets 2 August 2026 as the general date of application and lists no separate date for Article 50.[70] Article 111(4), which the Digital Omnibus on AI added, requires providers of such generative systems placed on the market before 2 August 2026 to take the necessary steps to comply with Article 50(2) by 2 December 2026. The AI Act Service Desk shows these articles in the EUR-Lex consolidated version of the act as at 27 July 2026.[71]

Research

Hirzle and colleagues' scoping review of extended reality (XR) and AI, presented at CHI 2023, screened 2,619 publications from 203 venues published between 2017 and 2021, reviewed 311 papers in depth, identified five main topics where XR and AI meet and proposed 13 research opportunities.[72] A systematic review of generative AI in XR by Ning and colleagues, an arXiv preprint posted in November 2025, examined 26 articles from 2023 to 2025.[73]

Paper Venue and year System Evaluation reported by the authors
Liu, Li and Gruteser, "Edge Assisted Real-time Object Detection for Mobile Augmented Reality"[74] MobiCom 2019 Object detection offloaded to an edge server, with rendering decoupled from offloading and fast on-device tracking, for AR and MR displays running at 60 fps On an Nvidia Jetson TX2, which has the same Tegra TX2 chip as the Magic Leap One, connected over Wi-Fi to a PC with a Titan Xp GPU that acted as the edge server: detection accuracy improved by 20.2-34.8% for object detection and human keypoint detection, with 2.24 ms of object tracking latency on the device
De La Torre et al., "LLMR"[75] arXiv preprint; version accepted at CHI 2024 Framework for creating and modifying interactive mixed reality scenes in Unity with large language models, using scene understanding, task planning, self-debugging and memory management The authors report that LLMR "outperforms the standard GPT-4 by 4x in average error rate"; usability study with 11 participants
Lee et al., "GazePointAR"[76] CHI 2024 Wearable AR voice assistant that uses eye gaze, pointing and conversation history to resolve pronouns in spoken questions 12-person lab study, then a first-person diary study of the refined system
Dogan et al., "Augmented Object Intelligence with XR-Objects"[77] UIST 2024 Open-source prototype that combines real-time object segmentation and classification with multimodal LLMs, without pre-registering objects, and attaches context menus to real objects Use cases and a user study
Srinidhi, Lu and Rowe, "XaiR"[78] IEEE ISMAR 2024 Split architecture with multimodal LLMs (GPT-4V and Ferret) on a server and 3D processing on a Magic Leap 2 headset, which streams audio and 640x480 images 15-participant study of a "cognitive assistant" application: over 90% accuracy in task guidance and 85% in AR content anchoring

See also

References

  1. ↑ Ronald T. Azuma (1997). "A Survey of Augmented Reality". Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385. https://doi.org/10.1162/pres.1997.6.4.355. Retrieved 2026-11-01.
  2. ↑ "Android XR". Android Developers. https://developer.android.com/develop/xr. Retrieved 2026-11-01.
  3. ↑ 3.0 3.1 "Integrate with the Gemini Live API for audio glasses and display glasses". Android Developers. 2026-05-19. https://developer.android.com/develop/xr/jetpack-xr-sdk/gemini-live. Retrieved 2026-11-01.
  4. ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 "Frequently Asked Questions". Meta for Developers. Meta Platforms. https://developers.meta.com/wearables/faq. Retrieved 2026-11-01.
  5. ↑ 5.0 5.1 5.2 "Foundation Models". Apple Developer Documentation. https://developer.apple.com/documentation/foundationmodels. Retrieved 2026-11-01.
  6. ↑ 6.0 6.1 6.2 6.3 "Gemini API using Firebase AI Logic". Firebase Documentation. 2026-10-27. https://firebase.google.com/docs/ai-logic. Retrieved 2026-11-01.
  7. ↑ 7.0 7.1 "Tracking and altering images". Apple Developer Documentation. https://developer.apple.com/documentation/arkit/tracking-and-altering-images. Retrieved 2026-11-01.
  8. ↑ 8.0 8.1 8.2 "Core AI". Apple Developer Documentation. https://developer.apple.com/documentation/coreai. Retrieved 2026-11-01.
  9. ↑ 9.0 9.1 "Verifying Device Support and User Permission". Apple Developer Documentation. https://developer.apple.com/documentation/arkit/verifying-device-support-and-user-permission. Retrieved 2026-11-01.
  10. ↑ 10.0 10.1 "capturedImage". Apple Developer Documentation. https://developer.apple.com/documentation/arkit/arframe/capturedimage. Retrieved 2026-11-01.
  11. ↑ Joe Rossignol (2026-09-09). "Apple Announces iOS 27 Release Date". MacRumors. https://macrumors.com/2026/09/09/apple-announces-ios-27-release-date/. Retrieved 2026-11-01.
  12. ↑ 12.0 12.1 12.2 "Implementing object tracking in your app". Apple Developer Documentation. https://developer.apple.com/documentation/visionos/implementing-object-tracking-in-your-app. Retrieved 2026-11-01.
  13. ↑ 13.0 13.1 "Accessing the main camera". Apple Developer Documentation. https://developer.apple.com/documentation/visionos/accessing-the-main-camera. Retrieved 2026-11-01.
  14. ↑ 14.0 14.1 14.2 14.3 "Building spatial experiences for business apps with enterprise APIs for visionOS". Apple Developer Documentation. https://developer.apple.com/documentation/visionos/building-spatial-experiences-for-business-apps-with-enterprise-apis. Retrieved 2026-11-01.
  15. ↑ "visionOS Group Lab - WWDC26". Apple Developer. 2026. https://developer.apple.com/videos/play/wwdc2026/8004. Retrieved 2026-11-01.
  16. ↑ 16.0 16.1 16.2 16.3 "ML Kit". Google for Developers. https://developers.google.com/ml-kit. Retrieved 2026-11-01.
  17. ↑ 17.0 17.1 17.2 "Overview of the ML Kit GenAI APIs". Google for Developers. 2026-10-29. https://developers.google.com/ml-kit/genai. Retrieved 2026-11-01.
  18. ↑ 18.0 18.1 18.2 18.3 "MediaPipe Solutions guide". Google AI Edge. 2026-10-28. https://developers.google.com/edge/mediapipe/solutions/guide. Retrieved 2026-11-01.
  19. ↑ 19.0 19.1 19.2 "Use ARCore as input for Machine Learning models". Google ARCore Developers. 2024-10-31. https://developers.google.com/ar/develop/java/machine-learning. Retrieved 2026-11-01.
  20. ↑ "Enhance your Android XR app with AI using Gemini APIs". Android Developers. 2026-10-28. https://developer.android.com/develop/xr/gemini. Retrieved 2026-11-01.
  21. ↑ 21.0 21.1 "Request hardware permissions for audio glasses and display glasses". Android Developers. 2026-10-28. https://developer.android.com/develop/xr/jetpack-xr-sdk/request-hardware-permissions. Retrieved 2026-11-01.
  22. ↑ 22.0 22.1 "Use a projected context to access hardware on audio glasses and display glasses". Android Developers. 2026-10-28. https://developer.android.com/develop/xr/jetpack-xr-sdk/access-hardware-projected-context. Retrieved 2026-11-01.
  23. ↑ Scott Hayden (2025-02-13). "Google Confirms Android XR Allows Passthrough Camera Access: an Important Tool for MR Devs". Road to VR. https://roadtovr.com/google-android-xr-passthrough-camera-access/. Retrieved 2026-11-01.
  24. ↑ "ARCore for Jetpack XR". Android Developers. https://developer.android.com/jetpack/androidx/releases/xr-arcore. Retrieved 2026-11-01.
  25. ↑ "Jetpack SceneCore". Android Developers. https://developer.android.com/jetpack/androidx/releases/xr-scenecore. Retrieved 2026-11-01.
  26. ↑ "XR Projected". Android Developers. https://developer.android.com/jetpack/androidx/releases/xr-projected. Retrieved 2026-11-01.
  27. ↑ 27.0 27.1 27.2 27.3 "AI Building Blocks - Overview". Meta Horizon OS Developers. 2026-07-08. https://developers.meta.com/horizon/documentation/unity/unity-ai-building-blocks-overview/. Retrieved 2026-11-01.
  28. ↑ 28.0 28.1 "Explore a New Era of Mixed Reality with the Passthrough Camera API". Meta for Developers. 2025-04-30. https://developers.meta.com/vr/blog/new-era-mixed-reality-passthrough-camera-api-machine-learning-computer-vision/. Retrieved 2026-11-01.
  29. ↑ 29.0 29.1 "Passthrough Camera API Overview". Meta Horizon OS Developers. 2026-10-07. https://developers.meta.com/horizon/documentation/unity/unity-pca-overview/. Retrieved 2026-11-01.
  30. ↑ 30.0 30.1 "Setup". Meta Wearables Developers. Meta Platforms. https://wearables.developer.meta.com/docs/develop/dat/getting-started-toolkit/. Retrieved 2026-11-02.
  31. ↑ 31.0 31.1 31.2 31.3 "Meta Connect Recap: How To Build For AI Glasses And Reach an Audience". Meta for Developers. 2026-09-24. https://developers.meta.com/blog/meta-connect-recap-ai-glasses/. Retrieved 2026-11-01.
  32. ↑ 32.0 32.1 "SnapML Overview". Snap for Developers. https://developers.snap.com/lens-studio/features/snap-ml/ml-overview. Retrieved 2026-11-01.
  33. ↑ 33.0 33.1 33.2 "Remote Service Gateway". Snap for Developers. https://developers.snap.com/spectacles/about-spectacles-features/apis/remoteservice-gateway. Retrieved 2026-11-01.
  34. ↑ 34.0 34.1 34.2 "Camera Module". Snap for Developers. https://developers.snap.com/spectacles/about-spectacles-features/apis/camera-module. Retrieved 2026-11-01.
  35. ↑ 35.0 35.1 David Heaney (2026-06-16). "Snap Opens Preorders For Specs, True AR Glasses Shipping This Fall For $2195". UploadVR. https://www.uploadvr.com/snap-specs-design-revealed-preorders-open-price/. Retrieved 2026-11-01.
  36. ↑ 36.0 36.1 "Raw camera access for WebXR". Web platform features explorer. https://web-platform-dx.github.io/web-features-explorer/features/webxr-camera. Retrieved 2026-11-01.
  37. ↑ 37.0 37.1 "WebXR Raw Camera Access Module". W3C Immersive Web Community Group. 2025-12-11. https://immersive-web.github.io/raw-camera-access/. Retrieved 2026-11-01.
  38. ↑ "Understand permissions for XR". Android Developers. 2026-10-28. https://developer.android.com/develop/xr/permissions. Retrieved 2026-11-01.
  39. ↑ "Permissions & Privacy". Snap for Developers. https://developers.snap.com/spectacles/permission-privacy/overview. Retrieved 2026-11-01.
  40. ↑ "Transparent Permission". Snap for Developers. https://developers.snap.com/spectacles/permission-privacy/transparent-permission. Retrieved 2026-11-01.
  41. ↑ 41.0 41.1 "Develop for the web on Android XR". Android Developers. 2026-08-31. https://developer.android.com/develop/xr/web. Retrieved 2026-11-01.
  42. ↑ "Understand the user's environment with the Scene Semantics API". Google ARCore Developers. 2026-09-04. https://developers.google.com/ar/develop/scene-semantics. Retrieved 2026-11-01.
  43. ↑ "Depth adds realism". Google ARCore Developers. 2026-09-04. https://developers.google.com/ar/develop/depth. Retrieved 2026-11-01.
  44. ↑ 44.0 44.1 "oculus-samples/Unity-PassthroughCameraApiSamples". GitHub. Meta Platforms. https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples. Retrieved 2026-11-01.
  45. ↑ "Changelog". Unity Documentation. https://docs.unity3d.com/Packages/[email protected]/changelog/CHANGELOG.html. Retrieved 2026-11-01.
  46. ↑ 46.0 46.1 46.2 46.3 "Foundation Models updates". Apple Developer Documentation. https://developer.apple.com/documentation/updates/foundationmodels. Retrieved 2026-11-01.
  47. ↑ "Build global-scale, immersive, location-based AR experiences with the ARCore Geospatial API". Google ARCore Developers. 2026-09-04. https://developers.google.com/ar/develop/geospatial. Retrieved 2026-11-01.
  48. ↑ 48.0 48.1 "Niantic Spatial". Niantic Spatial. https://www.nianticspatial.com/. Retrieved 2026-11-01.
  49. ↑ 49.0 49.1 "Snap to Launch New Lightweight, Immersive Specs in 2026". Snap Newsroom. 2025-06-10. https://newsroom.snap.com/launch-specs-2026. Retrieved 2026-11-01.
  50. ↑ "Gemini Live API overview". Google AI for Developers. 2026-10-06. https://ai.google.dev/gemini-api/docs/live. Retrieved 2026-11-01.
  51. ↑ David Heaney (2025-06-10). "Qualcomm Reveals Snapdragon AR1+ Chip For High-End Smart Glasses". UploadVR. https://www.uploadvr.com/qualcomm-snapdragon-ar1-plus-smart-glasses-chip/. Retrieved 2026-11-01.
  52. ↑ "Dreaming It and Building It With New GenAI Suite". Snap AR. 2024-06-18. https://ar.snap.com/blog/genai-suite-lens-studio-5.0. Retrieved 2026-11-01.
  53. ↑ 53.0 53.1 "Overview". Snap for Developers. https://developers.snap.com/lens-studio/features/lens-studio-ai/overview. Retrieved 2026-11-01.
  54. ↑ "AI Editor". Effect House. TikTok. https://effecthouse.tiktok.com/learn/guides/ai-capabilities/ai-editor/. Retrieved 2026-11-01.
  55. ↑ "Snap Launches New Tools for SPECS Developers". Snap Newsroom. 2026-06-16. https://newsroom.snap.com/snap-launches-new-tools-for-specs-developers. Retrieved 2026-11-01.
  56. ↑ "Writing code with intelligence in Xcode". Apple Developer Documentation. https://developer.apple.com/documentation/xcode/writing-code-with-intelligence-in-xcode. Retrieved 2026-11-01.
  57. ↑ "Setting up coding intelligence". Apple Developer Documentation. https://developer.apple.com/documentation/xcode/setting-up-coding-intelligence. Retrieved 2026-11-01.
  58. ↑ "About Gemini in Android Studio". Android Developers. 2026-08-25. https://developer.android.com/studio/gemini/overview. Retrieved 2026-11-01.
  59. ↑ 59.0 59.1 "AI-Native Development". GitHub. Meta Platforms. https://github.com/meta/immersive-web-sdk/blob/main/docs/ai/index.md. Retrieved 2026-11-01.
  60. ↑ Google AI Edge Team (2024-09-04). "TensorFlow Lite is now LiteRT". Google Developers Blog. https://developers.googleblog.com/en/tensorflow-lite-is-now-litert/. Retrieved 2026-11-01.
  61. ↑ "Migrate to the Firebase AI Logic SDKs from the GA version of Vertex AI in Firebase SDKs". Firebase Documentation. 2026-10-28. https://firebase.google.com/docs/ai-logic/migrate-to-latest-sdk. Retrieved 2026-11-01.
  62. ↑ "Sentis overview". Unity Documentation. https://docs.unity3d.com/Packages/[email protected]/manual/index.html. Retrieved 2026-11-01.
  63. ↑ "Recording and playback introduction". Google ARCore Developers. 2026-09-04. https://developers.google.com/ar/develop/recording-and-playback. Retrieved 2026-11-01.
  64. ↑ "Recording and Replaying AR Session Data". Apple Developer Documentation. https://developer.apple.com/documentation/arkit/recording-and-replaying-ar-session-data. Retrieved 2026-11-01.
  65. ↑ 65.0 65.1 "Exploring object tracking with ARKit". Apple Developer Documentation. https://developer.apple.com/documentation/visionos/exploring_object_tracking_with_arkit. Retrieved 2026-11-01.
  66. ↑ 66.0 66.1 "App Review Guidelines". Apple Developer. 2026-10-16. https://developer.apple.com/app-store/review/guidelines/. Retrieved 2026-11-01.
  67. ↑ Sarah Perez (2025-11-13). "Apple's new App Review Guidelines clamp down on apps sharing personal data with 'third-party AI'". TechCrunch. https://techcrunch.com/2025/11/13/apples-new-app-review-guidelines-clamp-down-on-apps-sharing-personal-data-with-third-party-ai. Retrieved 2026-11-01.
  68. ↑ "AI-Generated Content". Play Console Help. https://support.google.com/googleplay/android-developer/answer/13985936. Retrieved 2026-11-01.
  69. ↑ "Article 50: Transparency obligations for providers and deployers of certain AI systems". AI Act Service Desk. European Commission. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-50. Retrieved 2026-11-01.
  70. ↑ "Article 113: Entry into force and application". AI Act Service Desk. European Commission. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-113. Retrieved 2026-11-01.
  71. ↑ "Article 111: AI systems already placed on the market or put into service and general-purpose AI models already placed on the market". AI Act Service Desk. European Commission. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-111. Retrieved 2026-11-01.
  72. ↑ Teresa Hirzle et al. (2023). "When XR and AI Meet - A Scoping Review on Extended Reality and Artificial Intelligence". CHI '23, ACM. https://doi.org/10.1145/3544548.3581072. Retrieved 2026-11-01.
  73. ↑ Xinyu Ning et al. (2025-11-05). "When Generative Artificial Intelligence meets Extended Reality: A Systematic Review". arXiv preprint. https://arxiv.org/abs/2511.03282. Retrieved 2026-11-01.
  74. ↑ Luyang Liu, Hongyu Li, Marco Gruteser (2019). "Edge Assisted Real-time Object Detection for Mobile Augmented Reality". MobiCom '19, ACM. https://doi.org/10.1145/3300061.3300116. Retrieved 2026-11-01.
  75. ↑ Fernanda De La Torre et al. (2024-03-22). "LLMR: Real-time Prompting of Interactive Worlds using Large Language Models". arXiv preprint (version accepted at CHI 2024). https://arxiv.org/abs/2309.12276. Retrieved 2026-11-01.
  76. ↑ Jaewook Lee et al. (2024). "GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality". CHI 2024, Article 408, ACM. https://doi.org/10.1145/3613904.3642230. Retrieved 2026-11-01.
  77. ↑ Mustafa Doga Dogan et al. (2024). "Augmented Object Intelligence with XR-Objects". UIST 2024, ACM. https://doi.org/10.1145/3654777.3676379. Retrieved 2026-11-01.
  78. ↑ Sruti Srinidhi, Edward Lu, Anthony Rowe (2024). "XaiR: An XR Platform that Integrates Large Language Models with the Physical World". IEEE ISMAR 2024, pp. 759-767. https://doi.org/10.1109/ISMAR62088.2024.00091. Retrieved 2026-11-01.
Retrieved from "https://vrarwiki.com/index.php?title=How_to_develop_AR_apps_with_AI&oldid=53373"
Contents