Computer vision

Reviewed 13 October 2026. XR vision tasks, calibration, local feature methods, geometric tracking, depth reconstruction, learned perception and the stated study limitations checked against cited original papers and official documentation. About review dates.

Computer vision uses images and video to estimate information about objects and scenes. In augmented reality, these estimates help align virtual content with the physical environment. For example, ARCore combines camera features with inertial measurements to estimate camera position and orientation, and detects surfaces on which an application can place virtual objects. Vision also supports hand tracking for interaction: the original MediaPipe Hands pipeline estimates hand landmarks from a single RGB camera for AR and VR applications.

Goals and challenges for computer vision

Object recognition identifies an object or its category; pose estimation determines position and orientation relative to a coordinate frame. Scene understanding can combine geometric reconstruction with semantic labels, such as floor, wall or chair.

Task	Estimated information	XR use and distinction
Object recognition and detection	Identity, category or image region	DenseFusion segments known objects before estimating their rotation and translation.
Positional tracking	Camera position and orientation over time	Rendering virtual content from the camera's changing viewpoint. A geometric plane estimate does not itself identify the surface as a particular table.
Depth sensing and reconstruction	Depth samples or a three-dimensional surface model	Occlusion and collisions between virtual content and observed surfaces.
Scene understanding	Semantic labels associated with pixels or mapped surfaces	SemanticFusion combines per-pixel class probabilities with a geometric map, allowing reconstructed surfaces to carry labels.

Registration and calibration

To register an overlay with the real scene, the system needs the relationship between image coordinates, the camera and the world. Camera calibration estimates intrinsic parameters, such as focal scaling and the principal point, and models lens distortion. Extrinsic parameters describe rotation and translation between coordinate frames. Zhang's planar calibration method observes a pattern at different orientations, obtains an initial solution and refines it by minimizing image reprojection error, with radial distortion included in the model.

Calibration extends to relationships between sensors. VINS-Mono estimates the camera's pose relative to the inertial measurement unit (IMU), along with inertial biases, within its initialization and estimation procedures.

Approaches for implementation of computer vision

Local image features

Feature detection locates image structures; descriptors encode their surrounding appearance for matching. Geometric tracking can use feature correspondences or image intensities directly.

Salient point and blob detection

A salient point, or interest point, is a local image location selected for its distinctive structure. Interest points can be located through extrema of scale-normalized image derivatives, including the Laplacian and determinant of the Hessian. Blob extraction can also use connected image regions: OpenCV's SimpleBlobDetector thresholds an image at several levels, extracts connected components and groups their centres to return blob locations and sizes.

ORB combines FAST corner detection with a rotation-aware binary descriptor derived from BRIEF. Its original implementation ranks candidate corners using a Harris measure, detects them at levels of an image pyramid, assigns orientation from an intensity centroid and describes a patch using selected intensity comparisons. Such features provide correspondences between views; a corner alone does not establish the identity of the physical object that contains it.

Scale-space methods

Scale-space methods examine image structure at different smoothing scales. In SIFT, Gaussian convolution produces a family of smoothed images. Differences between nearby Gaussian scales locate candidate keypoints, which are checked against spatial and scale neighbours. The pyramid also downsamples images between octaves. Gaussian filtering performs smoothing; enlargement, when used, is a separate resampling operation. Lowe's implementation initially doubles the image using linear interpolation.

SIFT assigns each keypoint a scale and orientation, then describes local gradients with orientation histograms. This allows matching across changes in image scale and rotation. Its construction reduces some illumination effects, but the original paper also examines failures as viewpoint distortion and noise increase.

Template matching methods

Template matching compares an image patch with candidate regions of another image. A basic implementation slides a fixed template across the source image and records a similarity or difference score, using measures such as correlation or squared differences. The result identifies a matching image region at the template's dimensions. Changes in viewpoint or deformation can misalign corresponding pixels; Lowe's paper contrasts this sensitivity of simple patch correlation with local gradient descriptors.

Edge and boundary detection

Edge detection identifies changes in image intensity that can provide information about contours and boundaries. Canny's method uses Gaussian smoothing and gradient maxima, with nonmaximum suppression and hysteresis thresholding. Strong responses seed contours, and connected weaker responses are retained above a lower threshold. The analysis balances edge detection against localization and multiple responses to the same edge. An edge map is an intermediate representation: Canny's paper describes its use as input to geometric and recognition systems, rather than as a complete object interpretation.

Geometric pose, odometry and SLAM

Visual odometry estimates camera motion; SLAM also builds a map for localization. Feature correspondences constrain poses and three-dimensional points; bundle adjustment refines them through reprojection error minimization. Loop closure permits correction of accumulated drift when revisiting mapped areas. ORB-SLAM3 uses ORB features, supports visual and visual-inertial operation, and can create and merge maps after tracking loss.

Visual-inertial odometry combines visual observations with IMU measurements through sensor fusion. Monocular geometric reconstruction without a metric reference leaves scale undetermined; inertial information can make scale observable, subject to initialization and sufficient motion. VINS-Mono aligns a visual reconstruction with inertial data and jointly estimates motion, biases and other states. Its mobile AR demonstration ran on an iPhone 7 Plus and placed a virtual cube on a plane estimated from visual features.

In an AR application, the estimated camera pose sets the viewpoint used to render virtual content. As the map is refined, estimated poses and planes can change. ARCore's anchors maintain the relationship between placed content and tracked environmental features during such updates.

Depth and surface reconstruction

A depth map associates image locations with estimated distance information. Depth can be inferred from images captured at different camera positions or supplied by a depth sensor. ARCore's Depth API uses depth from motion and merges available hardware depth data, such as time-of-flight measurements. Its documented uses include occluding virtual objects behind real surfaces, depth-based hit testing and collisions. Support depends on the device, and depth must be enabled in the application.

KinectFusion provides an original example of dense geometric reconstruction for AR. It integrates successive Kinect depth frames into a volumetric truncated signed distance function and tracks the camera by aligning the current depth observations with a surface predicted from that model, using iterative closest point alignment. The 2011 system demonstrated geometry-aware AR interactions with reconstructed surfaces. Its tracking can become underconstrained when a large planar surface fills most of the camera's view, illustrating that a dense depth image does not always determine every component of motion.

Learned hand, object and semantic perception

Learned perception can estimate information beyond local corners or edges. MediaPipe Hands uses a palm detector to obtain an oriented crop, followed by a landmark model that predicts 21 points, hand presence and handedness. The 2020 paper's landmarks contain image coordinates and depth relative to the wrist. They are 2.5D predictions from RGB input, rather than direct metric measurements from a depth sensor. During tracking, the previous landmarks provide the next crop; detection runs again when the hand is lost.

DenseFusion estimates the six-degree-of-freedom pose of known objects from RGB-D input. After segmentation, it combines colour-image features and depth-derived point-cloud features at corresponding pixels, predicts candidate poses and iteratively refines the result. The paper evaluates this method on YCB-Video and LineMOD and demonstrates robotic manipulation. Those results concern the studied objects and input conditions; they do not establish pose accuracy for arbitrary objects in an XR scene.

SemanticFusion combines a convolutional network's per-pixel class probabilities with an ElasticFusion geometric map. Bayesian updates accumulate semantic evidence on mapped surface elements across views. The authors reported improved segmentation over single-frame predictions in their office reconstruction experiment and in their NYUv2 evaluation. The latter retained 140 video sequences after excluding those with frame rates below 2 Hz. Its labels and experiments cover defined classes in indoor RGB-D scenes.

Limits in XR use

Visibility and imaging conditions affect the evidence available to an estimator. ARCore's environment guidance identifies textureless surfaces, dim or extremely bright lighting, transparent or reflective surfaces, and moving surfaces as limitations for surface understanding. Its depth documentation also warns that surfaces with few features can have imprecise depth. Hand occlusion and self-occlusion are explicit challenges in the MediaPipe Hands paper; the detector runs again when the landmark model's hand-presence score falls below a threshold.

Geometric estimation depends on the available motion and environmental evidence. VINS-Mono discusses weakly observable or degenerate conditions. ORB-SLAM3 includes measurement uncertainty and attempts relocalization or starts another map when tracking fails.

Learned systems can encounter domain shift when deployment images differ from their training or evaluation data. In a controlled image-classification study, Recht and colleagues found lower accuracy on newly collected ImageNet test images despite closely following the original collection process. That result demonstrates sensitivity to data distribution; it supplies no numerical error rate for XR tracking or hand interaction.

Latency matters because an estimate describes a particular moment. Tracking, communication, rendering and display delays contribute to the interval before the corresponding image appears. If the viewpoint or an object moves during that interval, virtual content can be misregistered. Azuma's AR survey explains this dynamic error and approaches such as prediction and synchronization of real and virtual image streams.