Visual-inertial odometry

Visual-inertial odometry (VIO) is a motion tracking technique that estimates the six degrees of freedom (6DoF) position and orientation of a moving device by fusing images from one or more cameras with measurements from an inertial measurement unit (IMU). The two sensor types complement each other: the camera tracks visual features through the environment to anchor the estimate against drift, while the IMU supplies fast, high-rate motion data and the metric scale that a camera cannot recover on its own.

VIO is the core method behind inside-out tracking in modern standalone VR headsets and behind the world tracking in smartphone augmented reality. It runs entirely on sensors carried by the device, so it needs no external base stations, beacons, or markers placed in the room.

How it works

A VIO system runs two streams of estimation at once and merges them. From the camera, it detects visually distinct points in each frame, often image corners, and tracks how those points shift between successive frames. The way the points move encodes how the camera has translated and rotated, which lets the system recover its change in location. This part alone is visual odometry.

From the IMU, the system reads acceleration and angular velocity many hundreds of times per second. Integrating those readings predicts how the device moved between camera frames. IMU data is accurate over short time intervals and during sudden, fast motion, but it drifts over longer intervals because small measurement errors accumulate without bound.

The fusion turns these weaknesses into strengths. The camera observes stable landmarks in the world and corrects the slow drift of the integrated IMU, holding the trajectory accurate over time. The IMU, in turn, bridges the moments when vision fails: during fast motion that blurs the image, during the gap between camera frames, and during stretches where the camera sees a blank wall or a dark room with no trackable features. Because the IMU keeps predicting motion through those moments, tracking does not break when the camera momentarily has nothing useful to look at.

The IMU also resolves a problem that vision alone cannot solve. A single camera (monocular) cannot recover the true metric scale of a scene; from images alone, a small object seen up close and a large object seen far away can look identical. Adding an IMU makes the metric scale observable, along with the roll and pitch angles relative to gravity, as long as the device undergoes some acceleration. This is why a monocular camera paired with a low-cost IMU is regarded as the minimum sensor suite for metric 6DoF state estimation.

Loosely coupled vs tightly coupled fusion

VIO systems combine the visual and inertial information in one of two ways.

Approach	How fusion happens	Trade-off
Loosely coupled	The vision pipeline and the IMU are processed separately to produce two independent pose estimates, which are then merged (for example by a filter) into a single result.	More modular and computationally cheaper, but discards information by treating the two sources as black boxes.
Tightly coupled	Raw visual measurements (feature observations) and inertial measurements are fed into a single estimator that solves for all camera and IMU states jointly.	More accurate and more robust, at higher computational cost.

Most modern high-accuracy systems are tightly coupled and use a technique called IMU preintegration, which summarizes many IMU samples between two frames into a single constraint so the fusion can run efficiently in real time. Tightly coupled methods are commonly built either on a filter, such as the multi-state constraint Kalman filter (MSCKF), or on nonlinear optimization over a sliding window of recent frames.

Relationship to visual odometry and SLAM

VIO sits between two related ideas: plain visual odometry and full SLAM (simultaneous localization and mapping).

Odometry estimates motion incrementally, pose after pose, optimizing at most over a short recent window of frames. Because it only ever integrates new measurements onto old ones, its error accumulates and the estimate drifts over long trajectories; the device has no memory that it has returned to a place it visited before. SLAM adds a persistent map and a loop-closure step on top of odometry. When the system recognizes a previously visited location, it closes the loop and performs a global optimization that snaps the whole trajectory and map back into consistency, correcting the accumulated drift. In short, SLAM can be described as visual-inertial odometry plus place recognition and global map optimization. In practice the line blurs, because the tracking front end of a VIO system and the front end of a visual-inertial SLAM system are largely the same; the difference is whether a reusable map and loop closure are maintained.

Use in VR and AR

VIO is the backbone of markerless inside-out tracking in current consumer VR and AR hardware, where it replaced the external-sensor outside-in tracking used by earlier systems.

Standalone VR headsets

Standalone headsets such as the Meta Quest line track themselves with Oculus Insight, the inside-out positional tracking system introduced with the Oculus Quest and Rift S, which is built on SLAM and visual-inertial fusion. The Quest carries four wide-field-of-view fisheye cameras on the corners of the headset plus an IMU. The IMU runs at a high rate, on the order of 500 Hz to 1 kHz, while the cameras run at about 30 frames per second; a fast IMU thread predicts motion between camera updates while a slower mapper thread builds a point cloud of corner features and runs visual-inertial bundle adjustment to keep the map consistent. The headset cameras also see the infrared LEDs on the controllers, and because the spacing of those LEDs is known, the system recovers each controller's pose in the same coordinate frame as the headset.

Smartphone AR

The major smartphone AR platforms are VIO systems. Apple's ARKit, introduced in iOS 11, uses visual-inertial odometry to combine the device's motion-sensing hardware with computer-vision analysis of the camera scene: it recognizes notable features in the image, tracks how their positions change across video frames, and compares that against the motion-sensor data to produce a high-precision estimate of the device's position and motion in 6DoF. Google's ARCore does the same under the name concurrent odometry and mapping (COM): it detects feature points in the camera image to compute its change in location and fuses that visual information with inertial measurements from the device's IMU to estimate the pose of the phone relative to the world over time.

Implementations

• Basalt
• VINS-Mono, an open-source tightly coupled monocular visual-inertial state estimator.