Object recognition

See also: Scene understanding, Fiducial markers and Feature detection

Object recognition is the computer vision problem of identifying and locating the objects in images or video. In a 2020 survey of deep learning for generic object detection, Li Liu and colleagues use the term for "the more general problem of identifying/localizing all the objects present in an image, subsuming the problems of object detection and classification", and note that the literature has no universal agreement on the definitions of such vision subtasks. The survey lists augmented reality (AR) among the applications of object detection. AR systems that augment a physical object need real-time registration of that object: they estimate its pose, its 3D position and 3D orientation relative to the camera, six degrees of freedom in all, and keep the estimate updated as the camera or the object moves.

According to the survey, the leading approach to object recognition before 1990 was based on geometric representations; later work used statistical classifiers on appearance features and then handcrafted local features such as SIFT. Since a deep convolutional neural network set a record for image classification accuracy in 2012, the research focus in most aspects of computer vision has been on deep learning. The organizers of BOP, a benchmark for 6D object pose estimation that they created in 2017, report that the accuracy of model-based 6D localization of objects seen during training has since improved by more than 50% on the benchmark's seven core datasets, from a score of 56.9 to 86.0. Since 2023 BOP has also evaluated objects not seen during training, which methods must onboard from their CAD models in at most five minutes on one GPU; the highest-scoring method for this task reached a score of 82.5 in 2024.

As of November 2026, AR platforms and SDKs offer two broad kinds of object recognition. Model-based tracking detects and follows one specific object prepared from a 3D model or scan of it, as in Apple's object tracking for visionOS and iOS and PTC's Vuforia Model Targets. Category detectors label objects from a fixed list of classes, as in Google's ML Kit and MediaPipe object detectors, the object tracking extension of Android XR, and models that developers run on camera frames from Meta's Passthrough Camera API. Vendor documentation describes limits for both kinds: model-based tools need rigid objects, ARKit's scan-based detection works better with textured objects than with plain or reflective ones, Vuforia's Model Targets do not support shiny surfaces, and detectors recognize only the classes they were trained on.

Reviewed 2 November 2026. Checked each claim against its cited source: the papers and preprints (DOI, arXiv and proceedings records, abstracts and open full texts); Apple, PTC, Google, Meta, Unity, Snap, Niantic Spatial, Microsoft and Wikitude documentation and announcements; the Khronos OpenXR registry; the WWDC24 and WWDC26 session transcripts; and the BOP benchmark site About review dates.

Definition

The survey by Liu and colleagues distinguishes several related tasks. Pose estimation and tracking, which AR systems use to register virtual content on an object, are the subject of surveys of model-based tracking and of pose estimation for AR.

Task	Output	Notes
Image classification	One or more class labels for the whole image	Determines which classes are present, without locating them
Object detection	A class label and a location, usually a bounding box, for each object instance	Detecting one particular object, such as the Eiffel Tower in the survey's example, is "essentially a matching problem"; generic detection finds instances, usually not seen before, of predefined categories
Semantic segmentation	A class label for every pixel	The survey calls generic object detection closely related to it; it does not separate different instances of the same class
Instance segmentation	A pixel mask for each object instance	Distinguishes different instances of the same class
6D object pose estimation	The 3D translation and 3D rotation of an object relative to the camera	In the BOP benchmark, the training data for each object is a texture-mapped 3D model or images of the object in known 6D poses
Object tracking	The object's identity or pose, updated from frame to frame	ML Kit's streaming mode, for example, gives each detected object a tracking ID across frames

Neighboring tasks have their own articles: markers in Fiducial markers; planes in Plane detection, and meshes and room semantics in Scene understanding; keypoint detectors and descriptors in Feature detection; people in Hand tracking, Face tracking and Body tracking; and localization of the device against a previously captured map in Visual positioning system.

Ronald Azuma's 1997 survey of AR identified registration and sensing errors as "two of the biggest problems in building effective augmented reality systems". Eric Marchand, Hideaki Uchiyama and Fabien Spindler's 2016 hands-on survey relates the alignment of virtual elements with the real scene to "a pose estimation or, equivalently, a camera localization process", and covers techniques such as Perspective-n-Point (PnP) solutions, homography estimation, keypoint matching and SLAM. Vincent Lepetit and Pascal Fua's 2005 survey of monocular model-based 3D tracking of rigid objects reviews approaches from the AR and robotics communities, from those based on point or planar fiducial marks to those that avoid engineering the environment by relying on natural features such as edges, texture or interest points.

History

Geometric and appearance-based methods

According to Liu and colleagues, early object recognition research used template matching and simple part-based models, focusing on specific objects whose spatial layout is roughly rigid, such as faces. Before 1990 the leading paradigm was based on geometric representations; the focus later moved toward statistical classifiers, such as neural networks, support vector machines and AdaBoost, working on appearance features.

In a 1987 paper, W. Eric L. Grimson and Tomas Lozano-Perez described how local measurements of positions and surface normals can be used to identify and locate overlapping objects. They modeled objects as polyhedra or polygons with up to six degrees of positional freedom relative to the sensors, examined all hypotheses about pairings between sensed data and object surfaces, and efficiently discarded inconsistent pairings using local constraints such as distances between faces and angles between face normals. Y. Lamdan and Haim J. Wolfson's geometric hashing (1988) was a general model-based recognition method for occluded scenes, applicable in 2D and 3D, with special attention to recognizing 3D objects in occluded scenes from 2D gray-scale images. Bernt Schiele and James L. Crowley represented the appearance of objects by the joint statistics of local neighborhood operators such as Gaussian derivatives or Gabor filters, which they described as "a new class of appearance based techniques". In their experiments the method identified over 100 objects in the presence of major occlusions, and they reported that the techniques have low complexity and run in real time.

Local invariant features

David Lowe's 1999 ICCV paper described an object recognition system based on local image features that are invariant to image scaling, translation and rotation and partially invariant to illumination changes and affine or 3D projection. Candidate matches found by nearest-neighbor indexing were verified with a least-squares solution for the model parameters, and recognition in cluttered, partially occluded images took under two seconds. His 2004 journal paper, which calls the approach the Scale Invariant Feature Transform (SIFT), reported that a typical image of 500x500 pixels gives rise to about 2,000 stable features, a number that depends on image content and parameter choices, and that detecting small objects in cluttered backgrounds requires at least three features to be correctly matched from each object for reliable identification. Its recognition approach matched features to a database by nearest-neighbor search, identified clusters of matches belonging to a single object with a Hough transform, and verified them with a least-squares solution for consistent pose parameters, with near real-time performance. Liu and colleagues write that handcrafted local invariant features became very popular starting with SIFT, and that progress on visual recognition tasks rested substantially on local descriptors such as Haar-like features, SIFT, Shape Contexts, HOG, LBP and region covariances, usually aggregated by simple concatenation or by feature pooling encoders such as the bag of visual words. In their account, handcrafted features dominated the period up to 2012.

Model-based tracking for augmented reality

Lepetit and Fua list visual servoing of robotic arms, AR systems "that require real-time registration of the object to be augmented" and head tracking among the applications of 3D tracking of rigid objects. Tom Drummond and Roberto Cipolla's 2002 framework for model-based 3D tracking combined graphical rendering with constrained active contour tracking in a wire-frame tracking system that ran in real time, at video frame rate (25 Hz) on standard hardware. It rendered an internal CAD model of the tracked object, using a binary space partition tree for hidden line removal, and the paper also presents a visual servoing system built with the framework. L. Vacchetti, Vincent Lepetit and Pascal Fua formulated tracking as local bundle adjustment and merged information from preceding frames with a small number of keyframes created during a training stage; they reported a real-time tracker that does not jitter or drift. At ISMAR 2004, I. Skrypnyk and David Lowe presented a markerless AR system that built a sparse metric model of the environment from distinctive natural features and tracked the camera against it, without camera pre-calibration, prior knowledge of the scene geometry, manual initialization or special markers.

For objects with little texture, Stefan Hinterstoisser and colleagues presented multimodal templates at ICCV 2011, demonstrated on the combination of an image and a dense depth map; the method worked in real time under heavy clutter, did not require a time-consuming training stage and could handle untextured objects. The LM (Linemod) dataset of 15 texture-less household objects, which the BOP site credits to Hinterstoisser and colleagues (ACCV 2012), is part of the benchmark's BOP-Classic-Extra group.

Deep learning

Liu and colleagues date the shift to deep learning to 2012, when Alex Krizhevsky and colleagues' AlexNet, a deep convolutional neural network, achieved record-breaking image classification accuracy in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). According to the survey, object detection methods since then have been dominated by deep networks. Joseph Redmon and colleagues' YOLO (CVPR 2016) framed object detection as a regression problem: a single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. The base model processed images in real time at 45 frames per second and a smaller Fast YOLO at 155 frames per second; compared with other detection systems of the time, the authors reported, YOLO made more localization errors but was less likely to predict false positives on background. Mask R-CNN (Kaiming He and colleagues, ICCV 2017) extended the Faster R-CNN detector with a branch that predicts a mask for each object instance, and ran at 5 frames per second.

For 6D pose, PoseCNN (Yu Xiang and colleagues, RSS 2018) estimated an object's 3D translation by locating its center in the image and predicting its distance from the camera, and its 3D rotation by regressing a quaternion representation. It added a loss function for symmetric objects and introduced the YCB-Video dataset of 21 objects in 92 videos with 133,827 frames. DenseFusion (Chen Wang and colleagues, CVPR 2019) processed color and depth separately and fused them into per-pixel features for known objects; with iterative refinement it achieved near real-time inference, outperformed earlier methods on the YCB-Video and LineMOD datasets in the authors' tests, and was used on a real robot to grasp and manipulate objects. Bowen Wen and colleagues' FoundationPose (CVPR 2024) estimates and tracks the pose of novel objects without fine-tuning, given either a CAD model or a small number of reference images; the authors attribute its generalization to large-scale synthetic training aided by a large language model, a transformer-based architecture and contrastive learning.

Commercial toolkits

Date	Event
July 2017	Wikitude SDK 7 added object recognition and tracking for objects such as toys, sculptures, architectural models, product packaging and industrial machines; developers created a target by taking multiple pictures of the object and uploading them to Wikitude Studio Manager
October 2017	PTC announced the upcoming release of Vuforia 7, whose Model Targets "recognize objects by shape, in contrast to existing methods that rely on detailed visual designs"
June 2018	Apple announced ARKit 2, "making it possible to detect 3D objects like toys or sculptures"; the announcement said LEGO AR City uses ARKit 2 features including 3D object detection
March 2020	Google described MediaPipe Objectron, real-time 3D object detection on mobile devices, with trained models for two categories, shoes and chairs
1 March 2023	Google ended support for several MediaPipe legacy solutions, including Objectron, Box tracking and KNIFT; others, such as the legacy Object detection solution, were upgraded to new MediaPipe solutions
20 May 2024	Microsoft ended Azure Object Anchors, a mixed reality service that automatically aligned 3D content with physical objects using existing 3D models and that supported HoloLens 2 during its preview
June 2024	Apple announced visionOS 2, saying that developers including Scandit AG and TeamViewer were using new APIs such as Barcode Scanning and Object Tracking
21 July 2024	PTC deprecated Vuforia's Object Targets; release 10.29.6 (20 January 2025) removed the option to create them in the Target Manager and pointed developers to Scanned Model Targets
21 September 2024	All Wikitude services were shut down, after an end-of-life process announced from August 2023; Wikitude pointed developers to Snapdragon Spaces
February 2025	Vuforia Engine 11.0.4 (20 February 2025) introduced an Enterprise plan with On-Prem Advanced Model Targets, "generated without cloud training, keeping CAD data local", and Step Check
December 2025	Meta XR Core SDK version 83.0 (10 December 2025) added an AI Building Blocks collection for runtime AI features such as Object Detection and large language model integration
June 2026	At WWDC26, Apple presented high-frame-rate object tracking and an extended training mode for visionOS 27, a choice between display-corrected and uncorrected metric-space object poses, and support for reference objects in ARKit on iOS 27

How it works

Known objects from 3D models

Apple's object tracking starts from a 3D model of the physical object in USDZ format, made with computer-aided design software using physically based rendering materials, or captured with the Object Capture feature of Reality Composer on an iPhone or iPad. Developers train a machine learning model on it in Create ML to obtain a reference object file (.referenceobject) unique to the object. Training runs locally on a Mac with Apple silicon and macOS 15 or later and can take a few hours, and reference objects trained with macOS 27 and Xcode 27 or later require iOS 27 or visionOS 27. Apple asks for a 3D model that is as photorealistic as possible, in effect a digital twin of the object, and as precise as possible in scale, because a scale mismatch makes the virtual content appear offset in the viewing direction, in front of or behind the object. Training options include the viewing angles to support (All Angles, Upright or Front, with restricted angles tracking more accurately), "objects to avoid" as negative examples of similar items, and several objects in one project. Apps use the resulting file through ARKit, RealityKit or Reality Composer Pro, and object tracking requires an iPhone with iOS 27 or later or an Apple Vision Pro with visionOS 2 or later. In its WWDC24 session, Apple said that object tracking "requires individual machine learning training for each target object", that objects should have a rigid appearance in both shape and texture and be "non-symmetrical, meaning that they have a distinct appearance from all views", and that the feature is most effective with objects that are mostly stationary.

ARKit on iOS has supported detection of real-world objects since iOS 12, using .arobject reference objects that developers create by scanning the object, for example with Apple's scanning sample app. Apple's guidance for scanning is an illuminance of 250 to 400 lux with the object well lit from all sides, a light temperature of around 6500 K (D65), similar to daylight, and a matte, middle-gray background. The resulting reference object contains only the spatial feature information ARKit needs to recognize the object, not a displayable 3D reconstruction, and ARKit adds an anchor when it detects the object. Apple says detailed, textured objects work better for detection than plain or reflective ones, that scanning and detection are optimized for objects small enough to fit on a tabletop, that rigid objects work better than ones that bend, twist, fold or otherwise change shape, and that detection does better under lighting similar to that of the scan.

PTC's Vuforia Engine recognizes Model Targets "based on the shape of the object", for objects ranging from home appliances and toys to vehicles, large-scale industrial equipment and architectural landmarks. A Model Target is made from a 3D CAD model or a 3D scan of an object that is geometrically rigid and has stable surface features; shiny surfaces are not supported. Targets are prepared in the Model Target Generator, which runs on Windows and macOS. A standard Model Target uses a Guide View: the user holds the device at a particular angle and distance from the object, usually by lining up an image of the object that the app draws, before tracking can begin. Advanced Model Targets come from a cloud-based deep learning training process and can be recognized and tracked, one object at a time, from anywhere in a recognition range of up to 360 degrees without the user manually aligning the view.

Category detectors

Category detectors report the class, location and confidence of objects of the types they were trained on. Google's ML Kit object detection and tracking API runs on the device and can detect up to five objects in an image, or only the most prominent one; Google says it is optimized for live detection and tracking of the most prominent object in the camera viewfinder and for detecting multiple objects in a static image. Its optional coarse classifier, which Google says is built for five categories and gives limited information, sorts objects into fashion goods, food, home goods, places and plants. Apps can instead use a custom LiteRT classification model, or use ML Kit as the front end of a visual search pipeline that passes detected objects to a cloud backend such as Cloud Vision Product Search. On Android the API requires API level 23 or above. The MediaPipe Object Detector task takes still images, decoded video frames or a live video feed and outputs a category, a probability score and a bounding box for each detected object. Its recommended EfficientDet-Lite0 model was trained on the COCO dataset, which Google describes as having 80 object labels; in Google's benchmark on a Pixel 6, the float32 version took 61.30 ms per image on the CPU and 27.83 ms on the GPU, and the int8 version 29.31 ms on the CPU.

On Android XR, the OpenXR extension XR_ANDROID_spatial_object_tracking tracks "a set of predefined physical objects without requiring an input dataset from the application" and reports their poses and extents. Its labels are keyboard, mouse, laptop and uncategorized. Because the extension depends on the geometry of the environment, apps need the SCENE_UNDERSTANDING_COARSE permission, which the documentation calls a dangerous permission. As of November 2026 the extension is not ratified by the Khronos Group; it replaces XR_ANDROID_trackables_object, which the Khronos OpenXR registry lists as deprecated by it.

In Lens Studio, Snap's SnapML, introduced in Lens Studio 3.0, lets developers bring their own machine learning models into Lenses, from PyTorch, TensorFlow, Roboflow and other frameworks compatible with ONNX, in ONNX or TFLite format; Snap says SnapML uses device-specific hardware acceleration when it is available. Snap provides an Object Detection template that comes with car and food detection models, and a Multi Object Detection template whose model detects seven classes: cat, dog, potted plant, TV, car, bottle and cup.

The documentation of Niantic Spatial's NSDK 3.17.0, which still uses the earlier Lightship naming, describes an Object Detection feature that returns semantically labeled 2D bounding boxes. For each box it processes the central square crop of the image, and its neural network makes an independent prediction for each of 206 categories, so the box around a cat is predicted as both "cat" and "animal". The documentation includes a person detection model card, covering people, human hands and human faces, which says the model is designed for entertainment purposes within an AR application. Niantic Spatial's migration guide says that NSDK 3.17 receives only critical bug fixes and security updates through February 2027, and that in NSDK 4.0 all class names beginning with Lightship were renamed to use an Nsdk prefix. Neither the feature list of the NSDK 4.1.0 documentation, which names depth estimation, occlusion, scene segmentation, meshing and model preloading, nor its AR Effects page includes object detection.

On Meta's headsets, the Passthrough Camera API gives apps access to the forward-facing cameras of the Meta Quest 3 and Meta Quest 3S "for the purpose of supporting Computer Vision and Machine Learning". It requires Horizon OS v74 or later, either the android.permission.CAMERA or the horizonos.permission.HEADSET_CAMERA permission, and the passthrough feature to be enabled. Meta lists an image capture latency of 20-40 ms, a data rate of 60 Hz and a maximum resolution of 1728x1728. Meta's sample for running models with Unity's Inference Engine requires Inference Engine 2.2.1, the Mixed Reality Utility Kit (MRUK) 85.0.0 or later and the camera and spatial data permissions. It uses a Yolov9t model with an extra non-maximum suppression layer, quantized to Uint8 and about 2.3 MB in size, trained with 80 classes. Unity's changelog lists version 2.2.0 (15 May 2025) as the first version of Inference Engine and records that version 2.4.0 (22 October 2025) renamed Inference Engine to Sentis in the package name and documentation. Meta's AI Building Blocks include an Object Detection block that sends camera images to a provider, such as a Unity Inference Engine provider or a Hugging Face provider, which returns bounding boxes, class labels and confidence scores; Meta describes providers for cloud, local and on-device backends.

From 2D detections to 3D positions

A 2D detector reports positions in image pixels, which an AR app converts into a 3D location. Google's ARCore guide describes passing ARCore's CPU camera image, VGA (640x480) by default, to a model such as ML Kit's; Google says the default stream incurs no additional cost because ARCore already uses it for world comprehension. The app converts the center point of a detection, or the center of its bounding box, from image-pixel to view coordinates with Frame.transformCoordinates2d(), estimates the pose at that point with Frame.hitTest(), and attaches an anchor. Meta's Quest sample places a 3D marker with the class name at the real-world position of each detected object when the user presses the A button, finding that position with the Environment Raycasting feature of MRUK, which uses the device's depth map data. The AI Building Blocks instead map detections to world space with depth data from DepthTextureAccess. Meta calls raycasting "an alternative and still valid solution" but says the building blocks do not use it because it can get imprecise and expensive when done each frame and for each point of the bounding box, and because the ray may hit the wrong point if the user's head moves between inference and output retrieval.

Xiang Li and colleagues (ISMAR 2020) combined the camera pose and scale from ARCore's visual-inertial odometry with an SSD MobileNet detector on Android phones, using them to correct image orientation, filter detections by size and keep an online semantic map. On image sequences from 12 room-scale AR sessions, they reported a 12% accuracy improvement for generic object detectors. For Objectron, Google built a data pipeline on AR session data from ARCore and ARKit and reported that combining real-world data with AR synthetic data, made by placing virtual objects into scenes that have AR session data, increased accuracy by about 10%. The resulting pipeline computed oriented 3D bounding boxes of objects in real time on mobile devices.

Detecting, then tracking

Several documented pipelines run a detector at intervals and track objects between detections. Google's Objectron ran its model inference only once every few frames and tracked the boxes in between; when a new prediction arrived, it was merged with the tracking result based on the overlap area. Google reported 26 frames per second on an Adreno 650 mobile GPU. ML Kit's streaming mode runs with low latency and assigns tracking IDs, but may produce incomplete results, such as unspecified bounding boxes or category labels, on its earliest invocations, and may need to process 30 or more frames, depending on device performance, before it detects an object. Google's ARCore guide recommends against running a model on every incoming frame and suggests running object detection at a low frame rate instead. Since v85, Meta's Object Detection building block has a detectEveryNFrames setting from 0 to 120, where 1 runs detection on every frame and 0 runs it only when triggered manually through CallInference().

Apple's object tracking delivers pose updates at a lower frequency by default, which consumes less power and which Apple says works well for stationary objects. For moving or handheld objects, such as surgical instruments or power tools, apps set highFrameRateTrackingEnabled on visionOS or assign the object to trackingObjects on iOS to receive a pose update on every frame; Apple recommends training such objects in the extended mode, which it says takes significantly longer than the standard mode. On iOS 27, trackingObjects tracks objects at the full frame rate of the session's video format. Apple says high-frame-rate tracking significantly increases power consumption and processing load, and directs apps to use detectionObjects for mostly stationary objects. Only .referenceobject files work with trackingObjects; .arobject files work only with detectionObjects, and a session cannot use both formats.

Datasets and benchmarks

Dataset	Year	Contents	Main use
Microsoft COCO	2014	328,000 images with 2.5 million labeled instances of 91 object types	Detection and segmentation
LM (Linemod)	2012	15 texture-less household objects with discriminative color, shape and size	6D pose; part of BOP's BOP-Classic-Extra group
T-LESS	2017	30 industry-relevant objects without significant texture, recorded with three sensors; about 39,000 training and 10,000 test images per sensor	6D pose of texture-less objects; one of the seven BOP-Classic-Core datasets
YCB-Video	2018	21 objects in 92 videos with 133,827 frames	6D pose; introduced with PoseCNN; one of the seven BOP-Classic-Core datasets
Objectron	2021	4 million annotated images in 14,819 videos of nine categories, with AR session metadata such as camera poses, sparse point clouds and planes; collected in 10 countries on five continents	3D object detection; introduced a 3D intersection over union metric
HOT3D	2024	Over 833 minutes (more than 3.7 million images) of 19 subjects handling 33 rigid objects, recorded with Project Aria glasses and a Meta Quest 3; ground-truth poses from a motion-capture system with small optical markers (described in a preprint)	Egocentric hand and object tracking; part of BOP-H3

The BOP benchmark, presented at ECCV 2018 by Tomáš Hodaň and colleagues, evaluates 6D pose estimation of a rigid object from a single RGB-D input image. It brought eight datasets into a unified format, proposed a pose-error function that deals with pose ambiguities, and evaluated 15 recent methods; methods based on point-pair features performed best, outperforming template matching methods, learning-based methods and methods based on 3D local features. Its online evaluation system is open for continuous submission of new results. The 2023 challenge added objects unseen during training, which methods had to onboard from their 3D models in at most five minutes on one GPU. According to the organizers' preprint, the top-ranked 2023 method for unseen objects (GenFlow) reached the accuracy of the top-ranked 2020 method for seen objects (CosyPose), although it was noticeably slower.

The 2024 challenge introduced model-free tasks, in which objects are onboarded from reference videos without 3D models, a 6D detection task in which the identities of the visible objects are not given, and the BOP-H3 datasets recorded with high-resolution sensors and AR/VR headsets. The organizers' preprint reports that on the BOP-Classic-Core datasets the top-ranked 2024 method for model-based 6D localization of unseen objects (FreeZeV2.1) was 22% more accurate than GenFlow and only 4% behind the top-ranked 2023 method for seen objects (GPose2023), although significantly slower (24.9 against 2.7 seconds per image). The organizers describe Co-op as a more practical 2024 method for the task, at 0.8 seconds per image and 13% more accurate than GenFlow. They also found 2D detection of unseen objects still 35% behind that of seen objects, which made 2D detection the main bottleneck of pipelines for unseen objects. The organizers describe BOP-H3, made up of HOT3D, HOPEv2 and HANDAL, as focused on AR/VR scenarios. The 2025 challenge added the BOP-Industrial datasets (XYZ-IBD, ITODD and IPD), which focus on robotic bin picking, and a multi-view setup; its awards were published on 16 November 2025.

Platform and SDK support

The following table summarizes documented object recognition features as of November 2026.

Platform or SDK	Developer	Approach	Requirements and status
ARKit object detection (.arobject)	Apple	Scans of real objects; ARKit adds an anchor on detection	iOS 12 or later; textured, rigid, tabletop-sized objects work better; .arobject files work only with detectionObjects
ARKit and RealityKit object tracking (.referenceobject)	Apple	Model trained in Create ML from a USDZ model	iPhone with iOS 27 or later, or Apple Vision Pro with visionOS 2 or later; training on a Mac with Apple silicon and macOS 15 or later; on visionOS only inside an immersive space. Adjusting tracking parameters requires an entitlement for APIs that Apple makes eligible for business use only
Vuforia Model Targets and Advanced Model Targets	PTC	CAD model or 3D scan; Advanced Model Targets trained in the cloud with deep learning	The Basic plan allows development with Model Targets, with a watermark and a creation limit of 20 Model Targets, but no app-store publishing, which needs the annual Premium plan or the Enterprise plan; the Enterprise plan adds On-Prem Advanced Model Targets and Step Check, both in beta as of November 2026
ML Kit object detection and tracking	Google	On-device detector with a coarse or custom classifier	Android (API level 23 or above) and iOS; up to five objects per image
ARCore with machine learning	Google	ARCore supplies camera images to ML Kit or cloud models and hit-tests the results	Described in Google's ARCore developer guide (updated October 2024)
MediaPipe Object Detector	Google	EfficientDet-Lite0 trained on COCO, or custom models	Image, video and live-stream modes; upgraded from the legacy Object detection solution, while support for the legacy Objectron, Box tracking and KNIFT solutions ended on 1 March 2023
XR_ANDROID_spatial_object_tracking	Google (Android XR)	OpenXR vendor extension that tracks predefined objects without an input dataset and reports their poses and extents; labels are keyboard, mouse, laptop and uncategorized	Requires the SCENE_UNDERSTANDING_COARSE permission; not ratified by the Khronos Group; replaces the deprecated XR_ANDROID_trackables_object
Passthrough Camera API with Unity Sentis or AI Building Blocks	Meta	Developer-supplied models on camera frames; Object Detection building block	Meta Quest 3 and Meta Quest 3S with Horizon OS v74 or later and a camera permission; AI Building Blocks collection added in Meta XR Core SDK version 83.0 (December 2025)
Lens Studio SnapML	Snap Inc.	Imported ONNX or TFLite models; object detection templates	Introduced in Lens Studio 3.0
NSDK Object Detection	Niantic Spatial	2D boxes with predictions for 206 categories	Documented for NSDK 3.17.0, which receives only critical bug and security fixes through February 2027; the NSDK 4.1.0 feature list does not include object detection
Wikitude object recognition	Wikitude	Targets created from photos of the object (SDK 7, 2017)	All Wikitude services shut down on 21 September 2024
Azure Object Anchors	Microsoft	Alignment of content to physical objects using existing 3D models; supported HoloLens 2 during the preview	Ended on 20 May 2024

Use in augmented reality

Apple lists uses for its object tracking that include verifying that a tool is held and positioned correctly during an assembly process, guiding people through an item's features, history or behaviors when they look at it, helping people troubleshoot household items and appliances with a virtual manual, and storytelling that makes collectables and toys come to life. In its WWDC26 session on object tracking, Apple demonstrated a handheld medical probe measuring the distance between vertebrae on a physical spine model and said such tracking could enable use cases such as "surgical training, home remodeling, or guided assembly"; the session presents the uncorrected metric-space pose as useful for measuring distances between tracked objects. Apple's object scanning documentation gives the example of a museum app that adds interactive 3D visualizations when the user points a device at a displayed sculpture or artifact. Google said in May 2018 that Google Lens now worked in real time, proactively surfacing information and anchoring it to the things in the camera view, using machine learning both on the device and on cloud TPUs. Google's ARCore ML Kit sample, written in Kotlin, uses ML Kit and the Google Cloud Vision API to classify objects in the camera view and attach a label to each object in the virtual scene.

Object recognition is used in Enterprise AR for guidance, training and inspection. PTC said in 2017 that Model Targets would enable a new class of AR content that "can replace traditional user manuals and technical service instructions", and its 2025 release notes describe Step Check as an "AI-powered feature that visually verifies whether a step in a procedure has been performed correctly". Microsoft described Azure Object Anchors, before the service's retirement, as aligning 3D content to physical objects without markers or manual alignment for mixed reality guides, training and 3D inspections; during its preview the service supported HoloLens 2. Apple's visionOS 2 announcement said Scandit AG and TeamViewer were using the Barcode Scanning and Object Tracking APIs "to help organizations further improve day-to-day operations".

Meta's Passthrough Camera API documentation lists use cases that Meta anticipates, including specially trained ML or CV models that identify specific objects an application can interact with, such as dumbbells or audio mixers, and a museum tour in which existing off-device LLMs give information about a painting the user looks at. The Spatial Scanner sample for the Meta Spatial SDK uses ML Kit for real-time on-device detection and labeling of objects and can invoke the Llama 3.2 11B Vision model as a cloud service for more in-depth details about a detected object. Meta calls parts of the sample proof-of-concept and notes that it supports invoking Llama only through AWS Bedrock, an option that has a cost; that its MediaPipe and OpenCV alternatives to ML Kit lack persistent tracking across video frames; and that using the camera has a performance and energy cost, so the sample runs the camera session and image reading on background threads to prevent stuttering on the main UI thread and closes the camera session once the user selects a detected object. Other AI techniques for AR apps are covered in How to develop AR apps with AI.

Use in virtual and mixed reality

Fully immersive virtual reality experiences hide the user's surroundings, and some platforms recognize input devices so that users can see them. Meta's Quest software update v37, announced in January 2022, added support for the Apple Magic Keyboard: once the keyboard is paired under the Experimental Features panel in Settings and hand tracking is enabled, Quest 2 shows a 3D representation of both the keyboard and the user's hands in the Home environment. Quest had shown the Logitech K830 keyboard since v28, using what Meta called "experimental Bluetooth-enabled keyboard tracking tech", and Meta said it would also release its Tracked Keyboard SDK as part of Presence Platform. Apple said that with visionOS 2, Vision Pro would reveal the user's physical Magic Keyboard even when the user is fully immersed in an Environment or app; the announcement does not describe how the keyboard is detected. The Android XR object tracking extension includes keyboards, mice and laptops among its labels.

Headset recordings also support research on recognizing objects from the wearer's viewpoint, known as egocentric vision. HOT3D was recorded with Project Aria, which its authors describe as a research prototype of light-weight AR/AI glasses, and with a Quest 3. The BOP organizers grouped it with HOPEv2 and HANDAL as BOP-H3, a set of datasets focused on AR/VR scenarios.

Research

Studies of object recognition for AR have measured how to split the work between a device and a server, how to combine detectors with AR tracking, and what headset hardware can run.

Study	Venue	Setup	Reported results
Glimpse (Tiffany Yu-Han Chen and colleagues)	SenSys 2015	Recognition on server machines, with an active cache of video frames for tracking on the device; Android smartphones and Google Glass over Verizon, AT&T and a campus Wi-Fi network	With hardware face detection, 96.4-99.8% precision for continuous face recognition; 75-80% precision for road signs, where continuous detection without Glimpse's techniques was "non-functional" (0.2-1.9%)
Luyang Liu, Hongyu Li and Marco Gruteser	MobiCom 2019	Detection offloaded to the edge, with rendering decoupled from offloading and fast tracking on the device, for AR and MR systems running at 60 fps	Detection accuracy improved by 20.2-34.8% for object detection and human keypoint detection; 2.24 ms of object tracking latency on the AR device
Xiang Li and colleagues	ISMAR 2020	ARCore visual-inertial odometry combined with an SSD MobileNet detector on Android phones	12% accuracy improvement for generic object detectors on 12 room-scale AR sessions
Mikołaj Łysakowski and colleagues	IEEE EDGE 2023	YOLOv8n running on Microsoft HoloLens 2 through the Barracuda library in Unity, without a server, with a 100 ms end-to-end target	Mean inference time of 89.84 ms at 160x160 input; the authors write that the results suggest the configuration can still detect all instances of the test object, a smartphone, at distances closer than 2.5 m
Silvia Zaccardi and colleagues	Sensors, 2023	Deep learning models, including YOLO and U-Net models, run on HoloLens 2 with Unity Barracuda and Windows Machine Learning (WinML)	Inference times from a few milliseconds to seconds; Barracuda significantly faster than WinML (p < 0.005)
PeriphAR (Yutong Ren, Arnav Reddy and Michael Nebeling)	CHI 2026	Gaze-based selection with cues on a simulated monocular AR display, emulated on a Meta Quest Pro; user studies with 32 and 12 participants; an end-to-end proof-of-concept prototype that segmented real objects in passthrough video with YOLO11n	In the prototype, capturing the passthrough image and rendering the enhanced target took 0.72-1.55 s on average across four real-world tasks, with segmentation about 25% of that time

Limitations

Object properties

Model-based tools need objects that keep their shape. Apple gives a pair of scissors as an example of an object that is challenging to track because it changes shape in use, and says rigid objects work better for ARKit detection than soft bodies or items that bend, twist, fold or otherwise change shape. Surface appearance also matters: ARKit detection works better with detailed, textured objects than with plain or reflective ones, Vuforia does not support shiny surfaces for Model Targets, and Apple suggests CAD models with physically based materials for objects that are entirely or partly transparent, shiny or reflective. Apple's WWDC24 session also asks for objects that are non-symmetrical, and the BOP site describes the T-LESS objects as exhibiting symmetries and mutual similarities in shape or size.

Lighting, viewpoint and similar objects

Apple says ARKit detection does better when lighting is similar to the conditions of the original scan, and that its object tracking is more accurate when training is restricted to the viewing angles an app needs. A standard Vuforia Model Target has to be viewed from a particular angle and distance before tracking can begin; Advanced Model Targets remove that need but are recognized one at a time. Niantic lists scale, lighting, viewpoint, occlusion, motion blur and flicker (jittering between predictions for temporally adjacent frames) as factors that apply to its object detection, and for person detection also subgroups such as geographical region, skin tone, gender and body posture. A tracker can also recognize a similar-looking object instead of the intended one, which Apple addresses with negative examples during training.

Fixed classes and generalization

Category detectors recognize only their trained classes. Meta's Yolov9t sample model "has been trained with 80 classes (not objects)", so monitors and TVs share the class TV_Monitor; cell phones are in most cases identified as TV_Monitor, Quest 3 controllers may be unrecognized or mislabeled as remote controllers, and the sample's bounding boxes do not perfectly align with the detected objects. Meta says the model's accuracy is not 100% because it was optimized to work on devices like Quest 3. For the AI Building Blocks, Meta warns of possible bounding box misalignment and gives a workaround of setting the passthrough camera resolution to 1280x960, which it says was previously the default. Niantic notes that the training datasets of its person detection model have not been audited for diversity and may present biases not surfaced by its benchmarks. Benjamin Recht and colleagues built new ImageNet test sets by closely following the original dataset creation process and found accuracy drops of 11-14% for ImageNet classifiers; they write that their results suggest the drops are caused not by adaptivity but by the models' inability to generalize to slightly "harder" images. In the BOP 2024 results, 2D detection was the main bottleneck of pipelines for objects not seen during training.

Compute, power and latency

Meta's documentation for its Inference Engine sample, updated in May 2026, says the Inference Engine runs on the main thread of a Unity app, where it affects other main processes such as the render pipeline, and that it currently does not use the NPU or hardware acceleration; it also notes that large generative models and LLMs do not perform well on devices like Meta Quest 3. On iOS, Apple says high-frame-rate tracking significantly increases power consumption and processing load. Sending images to a server adds network delay: the Glimpse authors report that object recognition accuracy drops when the latency between server and device is higher than a frame time, and Liu, Li and Gruteser write that the long latency of existing offloading techniques can significantly reduce detection accuracy because the user's view changes in the meantime. On visionOS, object poses are by default slightly altered to match the displayed camera images, which Apple says affects accuracy in absolute world coordinates; visionOS 27 adds the ARKit Coordinate Space Correction API, which can return the uncorrected pose in metric space.

Access, privacy and availability

Camera-based recognition depends on platform permissions. Quest apps need the android.permission.CAMERA or horizonos.permission.HEADSET_CAMERA permission, the Android XR extension needs the SCENE_UNDERSTANDING_COARSE permission, and adjusting visionOS object-tracking parameters requires an entitlement for APIs that Apple makes eligible for business use only. On visionOS, object tracking works only in an immersive space and fails silently in a window or volume. Niantic says that when its model is used in its SDK, inference runs only on the device and the image is not transferred off it, and its model card lists tracking individuals and identifying or recognizing individuals as out-of-scope uses; wider privacy issues of camera-based AR are covered in Privacy in virtual and augmented reality. Several services have closed: support for MediaPipe's legacy Objectron solution ended in March 2023, Azure Object Anchors ended in May 2024, and Wikitude shut down all its services in September 2024 and pointed developers to Snapdragon Spaces.

See also

• Meta Wearables Device Access Toolkit
• Spatial anchors
• Depth sensing
• Positional tracking
• Photogrammetry
• 3D scanning