Kalman filter
The Kalman filter is a recursive algorithm that estimates the state of a dynamic system, such as the position, orientation and velocity of a moving head, from a series of noisy measurements. Greg Welch and Gary Bishop of the University of North Carolina at Chapel Hill describe it as "a set of mathematical equations that provides an efficient computational (recursive) means to estimate the state of a process, in a way that minimizes the mean of the squared error".[1] The method was published by Rudolf E. Kalman in 1960, and in the early 1960s researchers at NASA's Ames Research Center adapted it for Apollo navigation studies.[2][3]
In virtual reality (VR) and augmented reality (AR), the filter is used for head and hand tracking. It fuses readings from inertial sensors with optical, magnetic, acoustic or camera measurements, and it predicts where the user's head will be when the next frame reaches the display, which reduces the visible effect of system latency. In a 2009 history of the technique, Welch wrote that at VR technical conferences "it would be unusual to see a paper on tracking that did not use some form of a Kalman filter, or draw comparisons to those that do".[4] The Kalman filter is not the only choice: the head tracking that Oculus VR described for the Oculus Rift Development Kit used a complementary filter for drift correction, and published comparisons have found simpler predictors that perform as well in some VR tracking tasks.[5][6]
How it works
The filter models a process whose state changes over time according to a linear stochastic difference equation and is observed through measurements that are a linear function of the state. Both the process and the measurements are corrupted by noise that is assumed to be independent, white and normally distributed. The two noise sources are described by covariance matrices, conventionally written Q (process noise) and R (measurement noise).[1]
Each cycle of the filter has two steps. The time update, or "predict" step, projects the current state estimate and its error covariance forward to the next time step. The measurement update, or "correct" step, computes a weighting called the Kalman gain, blends a new measurement into the predicted estimate, and updates the error covariance. Welch and Bishop compare the loop to a predictor-corrector algorithm for solving numerical problems.[1] Because every new estimate is built from the previous one, the filter does not need to store or reprocess the full measurement history. Welch and Bishop describe this recursive structure as making practical implementations "much more feasible" than a Wiener filter, which operates on all of the data for each estimate.[1] Welch notes that the filter can estimate past, present and future states, including state elements that are not directly observable in the measurements, and that the gain is optimal in the sense of minimizing the trace of the error covariance when the process, measurement and noise models are appropriate.[4]
The noise covariances are tuning parameters. R is usually measured before the filter is run, while Q is harder to determine because the process being estimated cannot be observed directly. Welch and Bishop give a VR example of adapting Q during operation: when tracking the head of a user in a 3D virtual environment, the magnitude of Q can be reduced when the user seems to be moving slowly and increased when the motion changes rapidly.[1]
Nonlinear variants
Head pose, camera projection and orientation are nonlinear, so tracking systems rarely use the basic linear filter unchanged.
- The extended Kalman filter (EKF) linearizes the process and measurement functions about the current mean and covariance, using their partial derivatives in a manner Welch and Bishop liken to a Taylor series.[1] The EKF grew out of work at NASA's Ames Research Center in the early 1960s, where Stanley Schmidt's group adapted Kalman's linear theory to nonlinear navigation problems.[3]
- The unscented Kalman filter (UKF) replaces linearization with the unscented transformation, which propagates mean and covariance information through nonlinear functions. Simon Julier and Jeffrey Uhlmann wrote in 2004 that the EKF is "probably the most widely used estimation algorithm for nonlinear systems" but is difficult to implement and tune and "only reliable for systems that are almost linear on the time scale of the updates"; they describe the unscented transformation as more accurate, easier to implement and of the same order of computation as linearization.[7]
- A complementary error-state Kalman filter estimates the errors in integrated inertial signals rather than the motion itself. Welch described this approach for hybrid inertial and optical head tracking in a 1995 technical report, and Eric Foxlin used it in several commercial trackers.[4]
History
Kalman wrote the paper while at the Research Institute for Advanced Study in Baltimore, with support from the U.S. Air Force Office of Scientific Research. It was presented at the ASME Instruments and Regulators Conference in spring 1959 and published in the Transactions of the ASME, Journal of Basic Engineering in 1960.[2] The paper re-examined the classical filtering and prediction problem, which Norbert Wiener had treated through the Wiener-Hopf integral equation, using the "state transition" method of analysing dynamic systems. Kalman showed that the formulation applies without change to stationary and nonstationary statistics, derived an equation for the covariance matrix of the optimal estimation error, and showed that the filtering problem is the dual of the noise-free regulator problem.[2]
According to a 1985 NASA technical memorandum by Leonard McGee and Stanley Schmidt, researchers at Ames Research Center recognized the filter's value shortly after publication, following a visit by Kalman in the fall of 1960. They were studying navigation for the circumlunar missions of the Apollo program, and the breakthroughs they needed to turn Kalman's theory into a practical tool included the extended Kalman filter, which linearized about the current best estimate of the state.[3] The University of Florida, where Kalman was a professor emeritus, wrote after his death on 2 July 2016 that his method "was quickly well-received and used in numerous aeronautical and military projects, including the Apollo Program", and that he received the National Medal of Science in 2009.[8] Welch and Bishop attribute the filter's wide later use, particularly in autonomous and assisted navigation, in large part to advances in digital computing.[1]
Applications in VR and AR
Tracking error is the main source of visual error in many VR and AR systems. Welch cites Richard Holloway's 1995 analysis of an AR surgical-planning system, which concluded that head-tracking error is the major cause of registration error and that latency is one of the primary causes of tracking error. Holloway's rule of thumb for an arm's-length AR task was that 1 ms of latency corresponds to about 1 mm of misregistration. Because latency cannot be reduced to zero, VR systems predict head motion, and Welch writes that it was this prediction problem, rather than filtering or sensor fusion, that led to the first uses of the Kalman filter in VR.[4]
Early head-motion prediction
The earliest published use of a Kalman filter in a VR context that Welch identifies is the 1988 master's thesis of Captain Robert Rebo at the U.S. Air Force Institute of Technology. Rebo built a head-mounted display prototype for the Air Force Super Cockpit project, tracked with a Polhemus magnetic tracker, and applied a Kalman filter to the tracker's six-degree-of-freedom estimates to predict head motion.[4] A 1989 assessment by Robert Albrecht, quoted by Welch, found that Rebo's predictor improved image stabilization for slower head movements but did not significantly correct lag at faster movements and let the image "swim" at the end of quick movements.[4]
Jiandong Liang, Chris Shaw and Mark Green published work in 1990-1991 on filtering the output of the Polhemus Isotrak, which they measured at 151 ms of delay. They used a low-pass filter for position jitter and a Kalman filter for orientation prediction, running four independent filters on the elements of the orientation quaternion with a 50 ms step to match the tracker's 20 Hz rate. At MIT, Martin Friedmann, Thad Starner and Alex Pentland used Kalman filters to predict the motion of tracked drumsticks for a virtual drum project, and, to limit overshoot in the predictions, chose among several Kalman filters running different motion hypotheses.[4]
Fusing vision and inertial sensors
In 1993 Ali Azarbayejani, Thad Starner, Brad Horowitz and Alex Pentland at MIT tracked a user's head with cameras and passive computer vision. Welch identifies it as the first VR work to apply a Kalman filter to low-level sensor measurements (image features) rather than to complete pose estimates, and the first to use an extended Kalman filter, which the nonlinear camera projection required. In 1994 Satoru Emura and Susumu Tachi at the University of Tokyo presented a Kalman filter that fused gyroscope measurements with Polhemus estimates to obtain a higher estimation rate and lower latency than the magnetic tracker alone.[4]
At UNC, Ronald Azuma and Gary Bishop added three rate gyroscopes and three linear accelerometers to an optical see-through HMD tracked by the university's ceiling-mounted infrared LED system, and used a Kalman filter to fuse the optical pose estimates with the inertial measurements and extrapolate head pose into the future. The work was driven by an AR-assisted amniocentesis project, where tracking delay caused visible misregistration between the needle and the virtual imagery. At SIGGRAPH 1994 they reported that on average "prediction with inertial sensors produces errors 2-3 times lower than prediction without inertial sensors and 5-10 times lower than using no prediction at all".[4]
SCAAT and the HiBall tracker
UNC's second-generation optoelectronic tracker, the HiBall, produced its first results on 12 April 1997. It replaced the batch photogrammetric pose solution of the earlier system with an extended Kalman filter that incorporated each single LED sighting as it arrived. Welch and Bishop presented the method at SIGGRAPH 1997 as single-constraint-at-a-time (SCAAT) tracking. By also estimating the location of each LED, the filter built a map of the ceiling beacons while tracking the user. Welch reports that the system produced estimates at up to 3 kHz with around 1 ms of latency, and that a commercial version was sold by 3rdTech as the HiBall-3100.[4]
Commercial inertial trackers
Eric Foxlin founded InterSense in 1996 after leading a head-tracker project at MIT, and Welch writes that virtually every InterSense product has used some form of Kalman filter. Foxlin's 1996 orientation tracker used a complementary separate-bias Kalman filter to fuse rate gyroscopes, inclinometers and a fluxgate compass, and led to the InertiaCube. His later hybrid inertial-acoustic system, named Constellation, used a complementary error-state filter with acoustic range measurements handled in SCAAT fashion and became the InterSense IS-600. A hybrid inertial and camera tracker developed with Leonid Naimark, which used printed encoded patterns attached to the ceiling, led to the IS-1200.[4] Welch wrote in 2009 that the most common current use of the Kalman filter in VR and AR tracking appeared to be fusing computer vision measurements with other sensors, primarily inertial ones.[4]
Visual-inertial odometry
Kalman filters are also used for visual-inertial odometry, which estimates motion from cameras and an IMU. The Multi-State Constraint Kalman Filter (MSCKF), published by Anastasios Mourikis and Stergios Roumeliotis at ICRA 2007, is an EKF-based algorithm for real-time vision-aided inertial navigation. Its measurement model expresses the geometric constraints that arise when a static feature is seen from several camera poses without adding the feature's 3D position to the filter state, so its computational cost is linear in the number of features.[9]
Prediction for remote rendering
Prediction matters again when rendering happens away from the headset. Serhan Gül and colleagues at Fraunhofer HHI designed a Kalman filter for head-motion prediction in a cloud-based volumetric video streaming system, where server-side rendering adds motion-to-photon latency that can cause registration errors in mixed reality. Using 14 head-motion traces recorded with a Microsoft HoloLens application and a constant-velocity model on position and quaternion orientation, they found that the Kalman filter predicted head orientation 0.5 degrees more accurately than an autoregression model for a 60 ms look-ahead time.[10]
Timeline
| Year | Researchers | Work | Use of the Kalman filter |
|---|---|---|---|
| 1988 | Robert Rebo (U.S. Air Force Institute of Technology) | Predictive HMD tracking for the Super Cockpit project | Head-motion prediction from Polhemus pose estimates[4] |
| 1991 | Jiandong Liang, Chris Shaw, Mark Green | Temporal-spatial realism with the Polhemus Isotrak | Orientation prediction with four quaternion filters[4] |
| 1993 | Ali Azarbayejani, Thad Starner, Brad Horowitz, Alex Pentland (MIT) | Camera-based head tracking | Extended Kalman filter on image features[4] |
| 1994 | Satoru Emura, Susumu Tachi (University of Tokyo) | Gyroscope and magnetic tracker fusion | Higher rate, lower latency estimates[4] |
| 1994 | Ronald Azuma, Gary Bishop (UNC) | Inertial and optical fusion for see-through AR | Fusion and head-pose prediction[4] |
| 1997 | Greg Welch, Gary Bishop (UNC) | HiBall tracker, SCAAT | Extended Kalman filter on single LED sightings[4] |
| 1996-2003 | Eric Foxlin (MIT, InterSense) | InertiaCube, IS-600, IS-1200 | Complementary error-state filters for inertial hybrids[4] |
| 2007 | Anastasios Mourikis, Stergios Roumeliotis | MSCKF | EKF-based vision-aided inertial navigation[9] |
| 2020 | Serhan Gül et al. (Fraunhofer HHI) | Cloud-rendered volumetric video on HoloLens | Head-motion prediction for remote rendering[10] |
Alternatives and comparisons
Several VR studies have compared the Kalman filter with simpler methods. In 2003 Joseph LaViola of Brown University presented predictive tracking algorithms based on double exponential smoothing and reported that, compared with Kalman and extended Kalman filter predictors, they ran approximately 135 times faster with equivalent prediction performance and simpler implementations.[6] In a separate 2003 study of head and hand orientation represented as quaternions, LaViola found that the unscented and extended Kalman filters performed equivalently, and concluded that the extra computation of the UKF and the quasi-linear nature of quaternion dynamics made the EKF the better choice for VR applications.[11]
For the Oculus Rift Development Kit, Steven LaValle and colleagues at Oculus VR corrected gyroscope drift with gravity and magnetic field measurements using a complementary filter, a choice they said was "motivated by simplicity of implementation and adjustment based on perceptual experiments". To reduce effective latency they developed constant-rate and constant-acceleration prediction methods that use angular velocity measured by MEMS gyroscopes at 1000 Hz, with simple smoothing filters on the velocity and acceleration estimates.[5] Welch observed that poor tracking performance has tended to be attributed to the Kalman filter itself rather than to the structure and parameters of the process and measurement models, or to whether the filter suited the measurements at all.[4]
See also
- Sensor fusion
- Predictive tracking
- Inertial measurement unit
- Visual-inertial odometry
- Simultaneous localization and mapping
- Motion-to-photon latency
- Positional tracking
References
- ↑ 1.0 1.1 1.2 1.3 1.4 1.5 1.6 Greg Welch, Gary Bishop (2004-03-01). "An Introduction to the Kalman Filter". University of North Carolina at Chapel Hill, Department of Computer Science, TR 95-041. https://www.cs.utexas.edu/~pstone/Courses/393Rfall15/readings/Welch+Bishop-TR-95.pdf. Retrieved 2026-09-27.
- ↑ 2.0 2.1 2.2 R. E. Kalman (1960). "A New Approach to Linear Filtering and Prediction Problems". Transactions of the ASME, Journal of Basic Engineering, vol. 82, no. 1. pp. 35-45. doi:10.1115/1.3662552. https://doi.org/10.1115/1.3662552. Retrieved 2026-09-27.
- ↑ 3.0 3.1 3.2 Leonard A. McGee, Stanley F. Schmidt (1985-11). "Discovery of the Kalman Filter as a Practical Tool for Aerospace and Industry". NASA Technical Memorandum 86847. NASA Ames Research Center. https://ntrs.nasa.gov/api/citations/19860003843/downloads/19860003843.pdf. Retrieved 2026-09-27.
- ↑ 4.00 4.01 4.02 4.03 4.04 4.05 4.06 4.07 4.08 4.09 4.10 4.11 4.12 4.13 4.14 4.15 4.16 4.17 4.18 4.19 Gregory F. Welch (2009-02). "HISTORY: The Use of the Kalman Filter for Human Motion Tracking in Virtual Reality". Presence: Teleoperators and Virtual Environments, vol. 18, no. 1. pp. 72-91. doi:10.1162/pres.18.1.72. https://doi.org/10.1162/pres.18.1.72. Retrieved 2026-09-27.
- ↑ 5.0 5.1 Steven M. LaValle, Anna Yershova, Max Katsev, Michael Antonov (2014). "Head Tracking for the Oculus Rift". 2014 IEEE International Conference on Robotics and Automation (ICRA). pp. 187-194. https://msl.cs.illinois.edu/~lavalle/papers/LavYerKatAnt14.pdf. Retrieved 2026-09-27.
- ↑ 6.0 6.1 Joseph J. LaViola Jr. (2003). "Double Exponential Smoothing: An Alternative to Kalman Filter-Based Predictive Tracking". 7th International Immersive Projection Technologies Workshop and 9th Eurographics Workshop on Virtual Environments. pp. 199-206. https://cs.brown.edu/people/jlaviola/pubs/kfvsexp_final_laviola.pdf. Retrieved 2026-09-27.
- ↑ Simon J. Julier, Jeffrey K. Uhlmann (2004-03). "Unscented Filtering and Nonlinear Estimation". Proceedings of the IEEE, vol. 92, no. 3. pp. 401-422. doi:10.1109/JPROC.2003.823141. https://doi.org/10.1109/JPROC.2003.823141. Retrieved 2026-09-27.
- ↑ "Remembering Rudolf E. Kalman (1930-2016)". Herbert Wertheim College of Engineering, University of Florida. 2016-07-07. https://www.eng.ufl.edu/news/ece/remembering-rudolf-e-kalman-1930-2016/. Retrieved 2026-09-27.
- ↑ 9.0 9.1 Anastasios I. Mourikis, Stergios I. Roumeliotis (2007). "A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation". 2007 IEEE International Conference on Robotics and Automation (ICRA). pp. 3565-3572. doi:10.1109/ROBOT.2007.364024. https://doi.org/10.1109/ROBOT.2007.364024. Retrieved 2026-09-27.
- ↑ 10.0 10.1 Serhan Gül, Sebastian Bosse, Dimitri Podborski, Thomas Schierl, Cornelius Hellge (2020). "Kalman Filter-based Head Motion Prediction for Cloud-based Mixed Reality". Proceedings of the 28th ACM International Conference on Multimedia. doi:10.1145/3394171.3413699. https://arxiv.org/abs/2007.14084. Retrieved 2026-09-27.
- ↑ Joseph J. LaViola Jr. (2003). "A Comparison of Unscented and Extended Kalman Filtering for Estimating Quaternion Motion". Proceedings of the 2003 American Control Conference. pp. 2435-2440. doi:10.1109/ACC.2003.1243440. https://doi.org/10.1109/ACC.2003.1243440. Retrieved 2026-09-27.