Inside-out tracking

See also: Markerless inside-out tracking‎, Positional tracking and Outside-in tracking

Introduction

Figure 1. A Windows Mixed Reality headset, an example of an HMD using markerless inside-out tracking via cameras visible on the front (image: www.wareable.com)

Inside-out tracking is a method of positional tracking commonly used in virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. It enables devices like head-mounted displays (HMDs) and motion controllers to determine their position and orientation in 3D space by using sensors located *on* the device itself, looking outward at the surrounding environment.

It contrasts with outside-in tracking, where external sensors (e.g., cameras or laser emitters) are placed in the environment to track sensors or markers located on the HMD or controllers. In inside-out tracking, the "eyes" (cameras or other sensors) are on the moving object, making the system inherently egocentric.

How it Works

An inside-out tracking system observes the external world from the perspective of the device being tracked. As the device moves, the sensors detect changes relative to the environment, allowing the system to calculate the device's updated position and orientation (six degrees of freedom - 3 translational, 3 rotational). This calculation typically happens in real-time, allowing the virtual environment to respond accordingly.

There are two main approaches:

Marker-based Inside-out Tracking

Figure 2. Early Lighthouse prototype, an inside-out tracking system with 2-dimensional barcodes as fiducial markers.

This approach relies on placing artificial markers, known as fiducial markers, in the environment. The sensors on the tracked device are designed to detect these specific markers.

• Mechanism: Cameras or other optical sensors on the device identify the known patterns or shapes of the fiducial markers (e.g., QR codes, specific geometric patterns, infrared (IR) LEDs). By analyzing the position, size, and orientation of these markers in the sensor's view, the system can calculate the device's pose relative to them.
• Limitations: Tracking only functions when the markers are within the sensor's field of view and are not occluded. Requires setup of the environment with markers.
• Example Concept: While not typically used for modern VR HMDs, the Nintendo Wii Remote utilizes a form of marker-based inside-out tracking. The IR camera in the remote tracks the position of IR LEDs located in the stationary Sensor Bar to determine where the remote is pointing (though primarily for orientation and relative pointing, not full 6DoF positional tracking).

Markerless Inside-out Tracking

Main article: Markerless inside-out tracking

This is the most common approach used in modern standalone VR/MR headsets. It uses computer vision techniques to track the device's position based on recognizing natural features in the surrounding environment, without requiring any artificial markers.

• Mechanism: Cameras on the HMD capture video of the environment. Sophisticated algorithms, often involving SLAM (Simultaneous Localization and Mapping) or VIO (Visual-Inertial Odometry), identify and track distinct features (corners, edges, textures) in the environment. As the headset moves, the system tracks how these features move across the cameras' views to calculate the device's motion. Data from onboard Inertial Measurement Units (accelerometers and gyroscopes) is typically fused with the visual data. The IMU provides high-frequency motion data and helps predict position during moments when visual tracking is temporarily lost (e.g., fast movements, poor lighting).
• Environment Mapping: SLAM-based systems often create and update a map of the environment as the user moves around, allowing the device to re-localize itself within a known space.

Advantages

• Simpler Setup: No need to install external sensors, base stations, or markers in the room.
• Portability: Systems are often self-contained, making them easier to move between different locations.
• Potentially Unlimited Tracking Volume: Tracking is limited by the environment the sensors can see, not by the placement of external hardware. The device can theoretically be tracked in any sufficiently featured area.
• Lower System Cost (Potentially): Eliminating external tracking hardware can reduce the overall cost of the VR system, although it increases the complexity and computational cost on the headset itself.

Disadvantages

• Controller Occlusion: Tracking of controllers can be lost if they move outside the field of view of the HMD's cameras (e.g., behind the user's back, too close to the HMD). Predictive algorithms using IMU data help mitigate this for short periods.
• Environmental Dependence: Tracking performance can degrade in environments with poor lighting, few visual features (e.g., blank walls), or highly repetitive textures. Reflective surfaces can also sometimes cause issues.
• Computational Load: Processing sensor data and running tracking algorithms (especially SLAM/VIO) requires significant processing power on the device itself (HMD or connected computer), potentially impacting battery life or requiring more powerful onboard processors.
• Drift: While modern systems are robust, markerless tracking can sometimes accumulate small positional errors over time (drift), potentially requiring occasional recalibration or re-localization, although environment mapping helps significantly.
• Initialization: The system needs to initialize and potentially map a new area before tracking becomes fully stable.

Examples of Devices and Platforms Using Inside-out Tracking

Many modern VR/AR/MR systems utilize markerless inside-out tracking:

• Meta Quest series (Oculus Quest, Quest 2, Quest 3, Quest 3S, Quest Pro) - Uses the "Oculus Insight" tracking system.
• Oculus Rift S - Also used Oculus Insight.
• Windows Mixed Reality platform headsets (e.g., HP Reverb G1 & G2, Samsung HMD Odyssey, Acer AH101, Lenovo Explorer).
• HTC Vive Cosmos (with standard faceplate), HTC Vive Focus series, HTC Vive XR Elite.
• Pico Neo series (e.g., Pico Neo 3, Pico 4).
• Microsoft HoloLens 1 and 2 (AR headsets using HPUs for processing).
• Valve Index (Uses inside-out cameras primarily for passthrough view, room setup scanning, and potentially some experimental features, but relies on Lighthouse (outside-in) for its primary HMD and controller tracking).
• PlayStation VR2 (Uses inside-out cameras for HMD and controller tracking).
• Apple Vision Pro (MR headset).
• Lenovo Mirage Solo (Used Google's WorldSense tracking).
• Magic Leap 1 and 2.
• Nintendo Wii Remote (non-VR example using marker-based inside-out tracking for pointing/orientation).