Oculus Insight

Oculus Insight is the inside-out positional tracking system that Oculus, now part of Meta, introduced in 2019 for the Oculus Quest and Oculus Rift S. It tracks the headset and the Oculus Touch controllers in full six degrees of freedom using cameras built into the headset, with no external base stations or sensors. It replaced Constellation, the earlier external-sensor system used by the original Oculus Rift.

Oculus Insight was announced at Oculus Connect 5, held September 26 to 27, 2018 in San Jose, California, where it was shown as the tracking system for the then-unannounced-by-name Quest. Both the Quest and the Rift S shipped with the system on May 21, 2019.

How it works

Oculus Insight is a form of Inside-out tracking, which places the cameras on the headset and has them look outward at the room rather than relying on cameras mounted around the play space. The Quest carries four wide-angle cameras on the front of the headset; the Rift S has five cameras, with two on the front, one on each side, and one on top.

The headset is located using Visual-inertial odometry (VIO) combined with simultaneous localization and mapping (SLAM). The cameras capture video of the surroundings, and computer vision algorithms pick out distinct visual features such as corners, edges, and textures, then track how those features shift across the frames to work out how the headset is moving. At the same time the system builds and updates a 3D map of the room, pinpointing landmarks so it can keep the headset positioned within that map as the user walks around. An inertial measurement unit (IMU) in the headset supplies high-frequency motion data from its accelerometer and gyroscope. Fusing the IMU with the camera data fills in the gaps during fast movement and keeps tracking stable in the brief moments when the visual features are hard to read.

On the Quest the tracking runs entirely on the headset's mobile Qualcomm Snapdragon processor, since the device is a standalone headset with no external computer. On the Rift S the same work is handled by the connected PC.

Controller tracking

The Oculus Touch controllers are tracked by the same cameras on the headset. Each controller carries a ring of infrared LEDs that the headset cameras detect, and the known arrangement of those LEDs lets the system recover the controller's position and orientation in 6DOF. This is combined with an IMU inside each controller and with motion-prediction algorithms that estimate where a controller is heading when it leaves the cameras' field of view, for example when a hand drops to the side or reaches behind the head.

This LED approach carries over the basic idea behind Constellation, where infrared LEDs on the hardware were watched by infrared cameras, but it moves the cameras onto the headset instead of mounting them around the room. Because the cameras travel with the user, there is no fixed tracking volume and no line-of-sight to break toward an external sensor. When the Touch controllers were redesigned for the Rift S and Quest, the infrared tracking rings were moved to the top of each controller so they faced up toward the headset cameras rather than out toward room-mounted sensors.

Replacement of Constellation

Constellation was an outside-in system: infrared LEDs were embedded in the headset and controllers, and one or more external Oculus Sensors, each an infrared camera, watched those LEDs from fixed positions to compute pose. That arrangement could be precise, but it required the user to place and cable the sensors, kept the play area inside the volume those sensors could see, and could lose tracking if the user blocked the line of sight to them.

By putting the cameras on the headset, Oculus Insight removed the external hardware entirely. Setup no longer involved mounting sensors, the tracked area was not bounded by where the sensors could reach, and room-scale tracking worked out of the box. The Quest was the first consumer Oculus headset to ship with built-in 6DOF inside-out tracking, and the Rift S brought the same system to PC-connected VR.

Relationship to Guardian

The cameras and the room map that Oculus Insight builds also feed the Guardian boundary system, which lets the user draw a play-space boundary and warns them before they reach the edge of it. Because the headset already tracks its position inside the mapped room, Guardian can tell when the user approaches or crosses the boundary and show a grayscale Passthrough view of the real surroundings from the same cameras. Facebook stated that the camera and Guardian data used for tracking was processed on the device and was not collected or stored by the company.

Accuracy and development

To tune the system, Oculus engineers installed OptiTrack motion-capture cameras, the kind used for visual effects work, in homes and offices and compared the OptiTrack measurements against the Oculus Insight output. From that comparison they refined the computer vision algorithms until tracking was accurate to within a millimeter, enough to register a subtle tilt of the head or a small movement of the hand. The team also recorded thousands of hours of video across rooms that varied in lighting, decoration, and size, and built automated systems that replayed that footage to flag any change in tracking performance.

The technology traces back to 13th Lab, a Swedish computer vision company whose SLAM work Facebook acquired in 2014, and to the Santa Cruz standalone prototype shown in 2016. Engineers including Oskar Linde, a co-founder of 13th Lab, and Joel Hesch led the work that became Oculus Insight, developed across teams in Zurich, Menlo Park, and Seattle. Later Meta headsets carried the lineage forward: the Meta Quest line continued to use inside-out tracking, and on the Quest Pro the controllers became self-tracking, each with its own onboard cameras, so they no longer depended on the headset seeing their LEDs.