Predictive tracking

See also: Terms and Technical Terms

See also: Tracking

Reviewed 13 October 2026. Pose forecasts and motion assumptions, original experiment conditions and results, and display-time API and rendering distinctions. About review dates.

Introduction

Predictive tracking estimates the future position or orientation of a tracked object, such as a head, hand, or controller. In virtual reality (VR) and augmented reality (AR), it can supply a pose for the time an image will be displayed, compensating for movement during the intervening latency. A forecast specifies a target time, rather than just the current pose.

Prediction compensates for some visible effects of delay without removing the processing or transmission time itself. Its accuracy depends on the measurements and motion model, and on how far the forecast extends. A high update rate does not establish low latency.

In optical see-through AR, the real scene remains directly visible while a delayed graphical overlay moves relative to it. Forecasting the viewpoint can reduce this dynamic registration error. It does not correct static errors caused by optical distortion, tracker calibration, mechanical alignment, or incorrect viewing parameters.

History and development

In 1994, Ronald Azuma and Gary Bishop described a see-through HMD system with position and orientation predictors. An estimator combined tracker and inertial measurements; a predictor extrapolated its state to the expected appearance time of the rendered scene.

Their 1995 analysis examined why prediction errors and high-frequency jitter grow with the horizon.

Joseph LaViola's 2003 work compared double exponential smoothing with Kalman-based pose predictors. A 2014 paper by Steven LaValle and colleagues documented constant angular rate and constant angular acceleration prediction for the Oculus Rift Development Kit, alongside its orientation estimation and drift correction.

Latency sources

The interval between movement and its displayed consequence is often called motion-to-photon latency. Prediction can change the observed delay during a movement while the underlying pipeline still takes time to execute.

Stage	Contribution to delay
Sensing	Sensor response and camera capture precede the availability of a measurement.
Transport	Moving measurements to the host, including over USB, takes time.
Estimation and smoothing	Sensor fusion and noise filtering require computation; smoothing can also alter the temporal response of the signal.
Simulation and rendering	The application processes input and renders the scene before an image can be displayed.
Queues and synchronization	Buffers and synchronization between asynchronous tasks can introduce variable delay, or latency jitter.
Display timing	A completed image still has to reach its display interval. Scanout and pixel response are distinct from the duration for which a pixel remains illuminated.

Reducing pipeline delay remains useful alongside prediction. A longer delay requires extrapolation farther beyond the available measurements, increasing the opportunity for the motion to depart from the assumed trajectory.

How far should it predict into the future?

The prediction horizon extends from the estimated state's time to the target time. Display compensation targets expected presentation time, accounting for measurement age and remaining work. Different sensing paths can require different look-ahead intervals to depict a head and a camera-tracked hand at the same scene time.

Azuma and Bishop showed that a polynomial predictor with perfect inputs develops increasing error as the interval and input-motion frequency increase. Extrapolation can also magnify measurement noise.

In the 1994 AR prototype, timing error caused registration error, so the system synchronized clocks and estimated the horizon each rendering iteration.

Common prediction algorithms

Prediction methods differ in how they estimate motion and propagate it beyond the latest measurement. Filtering, forecasting, and sensor fusion can share a mathematical model while performing different tasks.

Method	Forecast and assumptions
Constant velocity extrapolation	For a translational position vector p, estimated velocity v, and horizon h, the forecast is p(t+h) = p(t) + v(t)h. It assumes velocity remains constant over h.
Constant acceleration extrapolation	With estimated acceleration a, the forecast becomes p(t+h) = p(t) + v(t)h + (1/2)a(t)h2. It assumes acceleration remains constant over h. Adding an acceleration term changes the motion assumption; it does not guarantee a better forecast.
Kalman filter and extended Kalman filter	An estimator combines a dynamics model with noisy measurements; prediction propagates its state to the requested time. The 1994 AR system used linear Kalman filters for position and an extended Kalman filter for quaternion orientation.
Alpha-beta-gamma filter	Fixed gains correct estimated position, velocity, and acceleration using the discrepancy between predicted and observed position. Gain selection balances noise smoothing against response to a maneuver. Tenne and Singh analyzed this tradeoff for constant-acceleration and circular target trajectories; that analysis is not a benchmark of commercial headset tracking.
Double exponential smoothing	LaViola's predictor models a changing linear trend, weighting recent observations more heavily. It predicts both position and orientation without directly measured velocity or acceleration. Its parameters and comparison results depend on the motion sequences and sampling conditions.
Learned predictors	A trained network maps a sequence of measurements to a future pose. Illahi and colleagues evaluated linear and LSTM models, including fusion of head pose, gaze, and controller pose, for remotely rendered VR. Representation, input modalities, and training choices affected the outcome.

Orientation and translation

The vector formulas require consistent units and coordinate frames: with h in seconds, velocity uses distance/s and acceleration uses distance/s².

Three-dimensional orientation uses rotations, such as unit quaternions, composed through multiplication rather than componentwise translational extrapolation. The rendered viewpoint uses the inverse of the tracked camera-to-world orientation.

For a sensor-to-world orientation, sensor-frame increments multiply on the right; world-frame increments on the left.

An inertial measurement unit also does not measure position directly. Recovering translation from acceleration requires integration and is sensitive to errors. Position tracking can use external observations, for example optical measurements.

Constant linear velocity is not observable from gyroscope and accelerometer measurements alone.

Implementation considerations

A predictor needs a usable state estimate. Calibration, initial registration, integration, and correction of accumulated drift are separate tracking problems. An accelerometer responds to gravity and motion, while magnetic measurements can be disturbed by nearby materials. Treating either as an ideal reference can corrupt the orientation estimate supplied to a predictor.

Sampling rate should be reported for the actual device and signal. In the Rift Development Kit studied in 2014, sensor observations were reported at 1000 Hz, but magnetometer readings changed at 220 Hz. The authors smoothed derivative estimates and shortened prediction for slower rotations to limit visible noise. This example does not establish a sampling-rate range for all headsets or controllers.

Display-time prediction in OpenXR

OpenXR provides a concrete workflow. In specification 1.1.63, xrWaitFrame returns XrFrameState.predictedDisplayTime, the anticipated midpoint of the next application-generated frame's display interval. The application calls xrBeginFrame before graphics work, uses its target display time for simulation, assigns it to XrViewLocateInfo.displayTime, calls xrLocateViews, renders with the returned views, and submits the corresponding time in XrFrameEndInfo.displayTime through xrEndFrame.

XrTime uses monotonic nanoseconds from a runtime-chosen epoch, not wall-clock time. Unless otherwise specified, zero and negative values are invalid. Repeated view queries for the same target time can return updated predictions as more measurements arrive. Applications must check orientation and position validity flags before reading those fields; valid data can be inferred without active tracking. The specification accepts future times subject to type limits, while warning that more distant predictions may be less accurate. This API contract does not prescribe one prediction algorithm or guarantee its accuracy.

Applications across systems

The 1994 see-through AR prototype evaluated prediction during walking, head rotations, and combined translation and rotation. It compared forecasts with tracker measurements and screen-space registration error.

Remotely rendered VR adds a use for forecasts at the server: predicting the viewer pose before a remotely generated frame reaches the headset. The 2023 learned-prediction study evaluated this task at a 100 ms horizon using OpenNEEDs data from 44 participants exploring two VR environments, with participants 2, 37, and 41 held out for testing.

Challenges and limitations

A forecast can overshoot when motion stops or changes direction. In a controller latency experiment, virtual movement initially lagged, then caught up as prediction became effective; the researchers also observed overshooting at sudden stops. Such behavior means a small average delay during smooth motion does not describe every instant of the trajectory.

Kalman-based prediction depends on the state model and its uncertainty assumptions, as well as the measurements.

Pose prediction cannot by itself repair the other parts of AR registration. For example, inaccurate eye-to-tracker alignment or viewing parameters can misplace an overlay even when the user remains still. Optical tracking also depends on visibility, while magnetic and inertial methods have different limitations; prediction inherits the information available from the tracking system.

Research developments

Research includes multimodal learning and motion-dependent evaluation, but results remain tied to the tested data. In the 2023 study, adding gaze and controller data improved some LSTM forecasts; the benefit varied across models. Inference timings used an NVIDIA RTX 3080 Ti, so they do not establish performance on a standalone headset processor.

A July 2025 preprint by Zhong and colleagues evaluated error-state Kalman predictors using Quest 3 pose data sampled at 100 Hz through ALVR. It separated motion into predictability classes and simulated dropped measurements. The authors reported lower errors for their higher-order variants than their baseline filters, but noted that average trajectory-error metrics do not fully characterize jitter or perceptual stability.

Comparison with other tracking techniques

Technique	Relationship to predictive tracking
Sensor fusion and SLAM	Fusion combines measurements; SLAM estimates a moving sensor's location while building a map. Their state estimates can supply a predictor, but estimating location or maintaining a map is not itself a forecast for a future display time.
Late latching	Updates pose-dependent rendering constants after command submission and before the GPU uses them. The 2015 VR Direct presentation described this for timewarp constants. Late latching changes when pose information is consumed; the supplied pose can still be predicted.
Timewarp and asynchronous timewarp	Reprojection transforms an already rendered image using newer pose information. Michael Antonov's 2015 explanation describes orientation-only timewarp and a separate asynchronous thread that warps the latest completed frame. Rotation-only warping does not correct translation or animate independently moving objects; positional warping requires depth and can expose areas absent from the original image.
asynchronous spacewarp	Meta's PC SDK documentation describes a frame-rate smoothing method that uses previous frames, animation detection, and camera and head translation to estimate a new frame. This forecasts image content as well as using tracking information. The documentation distinguishes PC Asynchronous SpaceWarp from Application SpaceWarp for Quest.

Image correction and pose prediction can operate together. Reprojection has limits when a new viewpoint reveals previously hidden surfaces, while prediction can choose a viewpoint for the original rendering. Neither method guarantees correct images after arbitrary movement.

Evaluation and reported results

Prediction experiments should identify the horizon, motion sequence, error metric, and reference trajectory. The following results illustrate specific evaluations; their numerical values are not interchangeable headset specifications.

Study	Conditions and reference	Reported result
LaValle and colleagues, 2014	Rift Development Kit orientation; 20 ms horizon; three-second head-turn sequence with peak speed about 240 degrees/s. Predictions were compared with later gyro-integrated estimates, explicitly not independent ground truth.	Average / worst angular error: no prediction 1.46302 / 4.77900 degrees; constant rate 0.19395 / 0.71637 degrees; constant acceleration 0.07596 / 0.35879 degrees.
LaViola, 2003	Six roughly 20-second head and hand sequences from an InterSense IS900 in a CAVE. Tests used 70 or 180 Hz sampling, 50 or 100 ms horizons, filtered reference trajectories, added Gaussian noise, and parameters optimized for each dataset and setting.	Double exponential smoothing had similar RMS prediction error to the tested derivative-free KF/EKF predictors. Average paired position/orientation computation took 3.3360 microseconds versus 458.7803 microseconds on an AMD Athlon XP 1800+.
Warburton and colleagues, 2023	Physical and virtual controller translations were co-registered using 240 fps camera video and display colors. Tests used Vive, Rift CV1, Rift S, and Index; Unity/SteamVR; a Windows 10 PC with Ryzen 2600X/GTX 1060.	Mean delays across tested configurations were 21-42 ms at movement onset and 2-13 ms during continuous motion. Values describe this controller measurement and software path, not all head motion or application workloads.