Inside-out tracking

See also: Terms and Technical Terms

See also: Markerless inside-out tracking, Positional tracking and Outside-in tracking

The front of a Meta Quest 3, showing the outward-facing cameras used for inside-out tracking

Introduction

Figure 1. A Windows Mixed Reality headset, an example of an HMD using markerless inside-out tracking via cameras visible on the front (image: www.wareable.com)

Inside-out tracking is a method of positional tracking used in virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems. It lets devices such as head-mounted displays (HMDs) and motion controllers determine their position and orientation in 3D space using sensors located on the device itself, looking outward at the surrounding environment.

It contrasts with outside-in tracking, where external sensors (for example cameras or laser emitters) are placed in the environment to track sensors or markers located on the HMD or controllers. In inside-out tracking, the cameras or other sensors are on the moving object, making the system inherently egocentric.

Almost all standalone (all-in-one) VR headsets shipping in the mid-2020s use markerless inside-out tracking, including the Meta Quest line, PlayStation VR2, Apple Vision Pro, and the HTC Vive Focus series. The approach removed the external base stations and cameras that earlier consumer systems such as the Oculus Rift CV1 (Constellation) and the original HTC Vive (Lighthouse) required, which made room-scale VR far simpler to set up.

How it works

An inside-out tracking system observes the external world from the perspective of the device being tracked. As the device moves, the sensors detect changes relative to the environment, allowing the system to calculate the device's updated position and orientation (six degrees of freedom: three translational, three rotational). This calculation happens in real time so that the virtual environment can respond as the user moves.

There are two main approaches.

Marker-based inside-out tracking

Figure 2. Early Lighthouse prototype, an inside-out tracking system with 2-dimensional barcodes as fiducial markers.

This approach relies on placing artificial markers, known as fiducial markers, in the environment. The sensors on the tracked device are designed to detect these specific markers.

• Mechanism: Cameras or other optical sensors on the device identify the known patterns or shapes of the fiducial markers (for example QR codes, specific geometric patterns, infrared (IR) LEDs). By analyzing the position, size, and orientation of these markers in the sensor's view, the system can calculate the device's pose relative to them.
• Limitations: Tracking only functions when the markers are within the sensor's field of view and are not occluded. It requires setting up the environment with markers in advance.
• Example: While not typically used for modern VR HMDs, the Nintendo Wii Remote uses a form of marker-based inside-out tracking. The IR camera in the remote tracks the position of IR LEDs in the stationary Sensor Bar to determine where the remote is pointing (though primarily for orientation and relative pointing, not full 6DoF positional tracking).

Markerless inside-out tracking

Main article: Markerless inside-out tracking

This is the most common approach in modern standalone VR and MR headsets. It uses computer vision techniques to track the device's position by recognizing natural features in the surrounding environment, without any artificial markers.

• Mechanism: Cameras on the HMD capture video of the environment. Algorithms, usually based on SLAM (Simultaneous Localization and Mapping) or Visual-inertial odometry (VIO, also referred to as VIO), identify and track distinct features (corners, edges, textures) in the scene. As the headset moves, the system tracks how these features shift across the cameras' views to calculate the device's motion. Data from onboard Inertial Measurement Units (IMUs, containing accelerometers and gyroscopes) is fused with the visual data. The IMU provides high-frequency motion data and helps predict position during moments when visual tracking is temporarily lost (for example fast movements or poor lighting).
• Environment mapping: SLAM-based systems build and update a map of the environment as the user moves around, which lets the device re-localize itself within a known space. On Meta's Oculus Insight system the headset detects thousands of 3D points in the environment and uses them to compute the headset's position roughly every millisecond, building a point cloud as the user moves while simultaneously locating the headset within it.

Controller tracking

Tracking the HMD is only part of the problem. Most VR systems also need to track handheld controllers, which is harder for an inside-out system because the controllers are separate objects that frequently leave the headset cameras' view. The common solution is optical-inertial: the controllers carry a ring or cluster of infrared LEDs that the headset's outward-facing cameras detect, while an IMU inside each controller supplies high-rate motion data. When a controller moves out of the cameras' view (for example behind the back or above the head), the system relies on the controller's IMU to estimate its pose by dead reckoning until it returns to view. Meta engineers noted that the infrared LEDs in the Quest controllers change appearance as they move closer to or farther from the headset, which the tracking algorithm has to account for; the team validated its tracking against OptiTrack motion-capture cameras during development. PlayStation VR2 uses the same general scheme: four cameras on the headset track the environment, and each VR2 Sense controller carries a ring of IR LEDs that the headset cameras detect.

Some headsets avoid controllers entirely and track the user's hands directly with the same outward-facing cameras. The Apple Vision Pro, which launched in February 2024, ships without handheld controllers and is operated by eye tracking, hand gestures, and voice; it uses a dozen cameras and several other sensors, including six world-facing tracking cameras, to map the environment and track the hands.

Contrast with outside-in tracking

In outside-in tracking the sensing hardware is fixed in the room and the tracked device carries markers or receivers. The Constellation system used by the Oculus Rift CV1 placed infrared LEDs in a known pattern on the headset and controllers and used one or more external infrared cameras to determine pose with sub-millimeter accuracy and very low latency; extra sensors behind the user enabled 360-degree coverage. Because the reference cameras never move, outside-in systems have historically offered very stable, low-latency tracking, but they require mounting and calibrating external hardware and they suffer when the tracked object is hidden from the fixed cameras (occlusion). Inside-out tracking reverses the geometry: the cameras move with the user, so there is no external hardware to install, but the headset must do all of the computer-vision work itself and controller occlusion becomes the dominant tracking failure mode rather than body occlusion.

Advantages

• Simpler setup: No need to install external sensors, base stations, or markers in the room.
• Portability: Systems are self-contained, which makes them easier to move between locations.
• Large tracking volume: Tracking is limited by what the sensors can see, not by the placement of external hardware. Meta demonstrated Oculus Insight tracking across a 4,000 square-foot area with no external sensors.
• Lower system cost: Removing external tracking hardware can reduce the overall cost of a VR system, although it increases the complexity and computational load on the headset itself.

Disadvantages

• Controller occlusion: Controller tracking can be lost if a controller moves outside the field of view of the HMD's cameras (for example behind the user's back, or too close to the HMD). Predictive algorithms using IMU data mitigate this for short periods.
• Environmental dependence: Tracking quality can degrade in poor lighting, in spaces with few visual features (for example blank walls), or with highly repetitive textures. The Oculus Santa Cruz prototype was demonstrated in a carefully controlled, brightly lit room with no windows or mirrors.
• Computational load: Running SLAM/VIO requires significant processing on the device itself, which can affect battery life or require more powerful onboard processors.
• Drift: Markerless tracking can accumulate small positional errors over time (drift), which may require occasional recalibration or re-localization, although environment mapping reduces this.
• Initialization: The system needs to initialize and map a new area before tracking becomes fully stable.

History and adoption

Inside-out tracking has roots in robotics and in mobile AR, where SLAM was used to map spaces and place virtual content well before it reached consumer VR. Oculus (then owned by Facebook) publicly demonstrated inside-out tracking for VR with the "Santa Cruz" prototype at Oculus Connect 3 in October 2016; the prototype was a modified Oculus Rift with four onboard cameras and an IMU handling all tracking, plus on-board compute.

The first consumer headsets to ship with inside-out tracking for both the headset and the controllers were the Windows Mixed Reality (WMR) headsets, the first wave of which became available on October 17, 2017 from Acer, Dell, HP, and Lenovo, later joined by Samsung and Asus. WMR's controllers were criticized for tracking quality compared with contemporary Rift and Vive controllers, because they were tracked only by the two cameras on the front of the headset, so a controller lost positional tracking whenever it moved out of that camera view. Microsoft deprecated Windows Mixed Reality in December 2023 and removed it from Windows 11 with version 24H2 in October 2024, which left those headsets without OS support.

The technology reached the mass market with Meta's Oculus Insight, which shipped in the Oculus Quest and Oculus Rift S in 2019. Insight uses four ultra-wide-angle cameras plus IMU data and on-device SLAM to track the headset and two controllers simultaneously, which Meta described as accurate to the millimeter, all running on the Quest's mobile chipset. Since then, inside-out tracking has become the default for standalone headsets, including the Meta Quest 2, Quest 3, Quest 3S and Quest Pro, the HTC Vive Focus series and HTC Vive XR Elite, the Pico line, PlayStation VR2, and the Apple Vision Pro. AR headsets including Microsoft HoloLens 1 and 2 and Magic Leap 1 and 2 also use inside-out tracking.

Examples of devices and platforms using inside-out tracking

Many modern VR, AR, and MR systems use markerless inside-out tracking:

• Meta Quest series (Oculus Quest, Quest 2, Quest 3, Quest 3S, Quest Pro) using the "Oculus Insight" tracking system.
• Oculus Rift S which also used Oculus Insight.
• Windows Mixed Reality platform headsets (for example HP Reverb G1 and G2, Samsung HMD Odyssey, Acer AH101, Lenovo Explorer).
• HTC Vive Cosmos (with the standard faceplate), HTC Vive Focus series, HTC Vive XR Elite.
• Pico Neo series (for example Pico Neo 3, Pico 4).
• Microsoft HoloLens 1 and 2 (AR headsets).
• PlayStation VR2 (uses inside-out cameras for HMD and controller tracking).
• Apple Vision Pro (MR headset, controller-free).
• Lenovo Mirage Solo (used Google's WorldSense tracking).
• Magic Leap 1 and 2.
• Nintendo Wii Remote (non-VR example using marker-based inside-out tracking for pointing).