Sensor fusion

Sensor fusion is the practice of combining data from two or more sensors to produce an estimate that is more accurate, more stable, and more reliable than the estimate any single sensor could provide on its own. The idea rests on the fact that different sensors fail in different ways: where one is noisy, another is steady; where one drifts over time, another holds an absolute reference. By blending their outputs, a fusion algorithm keeps the strengths of each sensor while cancelling out the weaknesses.

In virtual reality and augmented reality hardware, sensor fusion is the technique that makes head and controller tracking feel instant and stable. Every modern headset reads several motion sensors at once, and none of them is trustworthy alone: a gyroscope is fast but drifts, an accelerometer is steady but noisy, a magnetometer gives an absolute heading but is easily disturbed, and cameras see the world but only at a relatively slow frame rate. Fusion merges these streams into one continuous, low-latency estimate of where the user is looking and where their hands are.

Why a single sensor is not enough

The need for fusion comes directly from the limitations of the individual sensors inside an IMU (inertial measurement unit). Each one measures something useful, and each one has a characteristic flaw.

Sensor	Measures	Strength	Weakness
Gyroscope	angular velocity (rate of rotation)	fast and smooth response to rotation, accurate over short intervals	the reading must be integrated over time, so small bias and noise accumulate into unbounded drift
Accelerometer	linear acceleration, including the constant pull of gravity	gives a stable absolute reference for "down," fixing pitch and roll	noisy, and during real movement it senses the user's own motion mixed in with gravity
Magnetometer	the local magnetic field, including the Earth's field	gives an absolute heading reference that can correct yaw	easily distorted by nearby metal, motors, and electronics

A gyroscope alone can report rotation quickly, but because its output is integrated to recover orientation, every tiny error builds up and the estimate slowly wanders away from the truth. This is gyroscope drift, and it is worst in yaw, the heading around the vertical axis, because nothing inside the gyroscope provides an absolute reference to pull it back. An accelerometer alone can find "down" by sensing gravity, but it cannot tell rotation about the vertical axis and its readings are corrupted whenever the device actually accelerates. A magnetometer alone can point toward magnetic north, but a headset packed with electronics is a hostile magnetic environment. Fusion exists because no one of these sensors solves the problem, while together they can.

The classic VR example: fusing the IMU

The most common use of sensor fusion in VR is orientation tracking, the job of working out how a headset or controller is tilted and turned. A headset IMU typically carries a three-axis gyroscope, a three-axis accelerometer, and sometimes a three-axis magnetometer. When all three are present the package is often called a 9-axis IMU, or a MARG (magnetic, angular rate, and gravity) array.

Fusion assigns each sensor the part of the task it does best:

• The gyroscope carries the fast, short-term rotation. Integrating its three-axis output gives a smooth, responsive, low-latency estimate of how the device has turned from moment to moment. This part is sometimes called dead reckoning, and on its own it drifts.
• The accelerometer corrects pitch and roll. Because the accelerometer senses the constant downward pull of gravity, it provides a stable estimate of which way is down, and that "down" vector pins the pitch and roll of the orientation to the true vertical, cancelling the part of the gyroscope's drift that would otherwise tip the horizon.
• The magnetometer corrects yaw. Gravity says nothing about heading, so the drift in yaw, the rotation around the vertical axis, cannot be fixed by the accelerometer. The magnetometer supplies an absolute heading from the Earth's magnetic field, and fusion uses it to pull the yaw estimate back toward true.

The result is an orientation estimate that updates as fast as the gyroscope and stays accurate as long as the accelerometer and magnetometer keep correcting it: fast where the gyroscope is fast, drift-free where the gyroscope drifts. An IMU enhanced with fusion software in this way is known as an attitude and heading reference system (AHRS).

A device that omits the magnetometer, using only a gyroscope and accelerometer, is a 6-axis configuration. Some VR designs deliberately leave the magnetometer out to avoid magnetic interference, accepting slow drift in heading as the price; that drift then has to be corrected another way, such as by the cameras in a positional tracking system.

Common fusion algorithms

Several well-known algorithms perform this blending. They range from a few lines of arithmetic to full statistical estimators, and they trade computational cost against accuracy.

Complementary filter

The complementary filter is the simplest and cheapest approach, and it captures the core intuition of fusion directly. It applies a high-pass filter to the gyroscope signal and a low-pass filter to the accelerometer signal, then adds them together. The high frequencies, the fast motion, come from the gyroscope, which is reliable over short intervals; the low frequencies, the steady long-term reference, come from the accelerometer, which does not drift. A single blending weight controls how quickly the accelerometer's correction is allowed to pull on the gyroscope's estimate. Because it is easy to implement and computationally light, the complementary filter is widely used where processing power is limited.

Kalman filter and extended Kalman filter

The Kalman filter is a recursive estimator that models the system's state and its uncertainty explicitly, predicting the next state from a motion model and then correcting that prediction with each new measurement. It was introduced by Rudolf E. Kalman in 1960 in the paper "A New Approach to Linear Filtering and Prediction Problems." Applied to an IMU, it tracks both the orientation and the slowly changing gyroscope bias, and it weights each sensor according to how much that sensor is currently trusted. A Kalman filter generally produces a more accurate estimate than a complementary filter, at the cost of greater computational complexity and harder parameter tuning.

The basic Kalman filter assumes the system is linear, which orientation tracking is not. The extended Kalman filter (EKF) handles this by linearizing the nonlinear motion and measurement models at each step, which makes it the standard tool for orientation and pose estimation. The EKF is also the workhorse of camera-and-IMU fusion described below, valued for its low memory use and fast, real-time operation, though linearization can make it less stable when the nonlinearities are strong.

Madgwick and Mahony filters

Two filters designed specifically for low-cost IMUs are widely used because they are nearly as cheap as a complementary filter yet much more capable. The Madgwick filter was published by Sebastian O. H. Madgwick in 2010. It represents orientation as a quaternion, which avoids the singularities (the "gimbal lock") of Euler angles, and it uses an analytically derived, optimised gradient-descent step to compute the direction of the gyroscope's error from the accelerometer and magnetometer readings. It works with both 6-axis IMU arrays and 9-axis MARG arrays, includes magnetic-distortion and gyroscope-bias compensation, and is deliberately cheap: a single update needs only 109 scalar operations for the IMU case and 277 for the MARG case, and it stays accurate at low sampling rates.

The Mahony filter comes from the 2008 paper "Nonlinear Complementary Filters on the Special Orthogonal Group" by Robert Mahony, Tarek Hamel, and Jean-Michel Pflimlin. It is built to get good attitude estimates from the noisy, biased output of typical low-cost IMUs. Where the Madgwick filter uses only a proportional correction, the Mahony filter uses a proportional-plus-integral controller to estimate and remove the gyroscope bias. Both are popular in small embedded systems, including drone flight controllers, for the same reason they suit headsets and controllers: high accuracy with very little computation.

Fusing cameras with IMUs

Orientation fusion gives the three rotational degrees of freedom, but it cannot tell where a headset is in a room. To recover position as well, modern headsets fuse the IMU with one or more cameras in a technique called visual-inertial odometry (VIO), which is the basis of markerless inside-out tracking.

The two sensor types complement each other almost perfectly. The cameras detect distinctive features in the environment and track how those features move across frames, which anchors the estimate to the real world and corrects the slow drift of the integrated IMU. The IMU, running hundreds of times per second, fills the gaps the cameras leave: it predicts motion in the time between camera frames, it keeps tracking through fast motion that blurs the image, and it carries the estimate through moments when the cameras see a blank wall with nothing to lock onto. The IMU also makes the true scale of the scene observable, which a single camera cannot recover on its own.

This camera-and-IMU fusion is usually done in one of two ways. In loosely coupled fusion the vision system and the IMU are processed separately into two pose estimates that are then merged, which is simpler but discards information. In tightly coupled fusion the raw feature observations and the inertial measurements are fed into a single estimator that solves for everything jointly, which is more accurate and more robust at higher computational cost. Tightly coupled systems are commonly built on an extended Kalman filter, such as the multi-state constraint Kalman filter (MSCKF), or on nonlinear optimization over a sliding window of recent frames.

Why fusion is essential for VR and AR

Sensor fusion is not a refinement in VR and AR; it is what makes head and controller tracking usable at all.

The first reason is latency. A headset has to update the rendered view almost the instant the user moves, because the gap between a real head motion and the matching change on the display, the motion-to-photon latency, is a leading cause of discomfort and simulator sickness in VR. Fusion gives the system the best of both timescales: the gyroscope's fast response keeps the estimate current at very low latency, while the slower absolute references keep it correct. Headset IMUs are read at high rates, on the order of hundreds of samples per second up to about 1 kHz, specifically so the fast part of the fused estimate stays tightly synchronized with real motion.

The second reason is drift. Without fusion, the gyroscope-only estimate would creep away from the truth within seconds, and the virtual world would slowly slide out from under the user. The accelerometer and magnetometer (for orientation) and the cameras (for position) supply the absolute references that hold the estimate locked to the real world over long sessions. Fusion is therefore what delivers tracking that is simultaneously fast, smooth, and drift-free, the combination rotational and positional tracking both depend on.

See also

• IMU
• Gyroscope
• Accelerometer
• Magnetometer
• Visual-inertial odometry
• Inside-out tracking
• Rotational tracking
• Tracking