Sensor

A sensor is a device that detects a physical property of its environment, such as motion, light, distance, temperature or magnetic field, and converts it into a signal that an electronic system can read. In measurement terminology a sensor is closely related to a transducer, which converts one form of energy into another; the common engineering definition, attributed to the Instrument Society of America, is a device that produces a usable output in response to a specified measurand. Most modern sensors output an electrical signal, either analog (a continuous voltage proportional to the measured quantity) or digital.

Sensors are central to virtual reality (VR), augmented reality (AR) and mixed reality (MR) hardware. A head-mounted display must know, many times per second, where the user's head is in space, where the controllers and hands are, how far away the surrounding walls and furniture are, and where the eyes are pointing. Each of these measurements comes from one or more sensors, and the headset combines their outputs through sensor fusion to produce a stable estimate of pose and environment.

Sensor categories used in VR and AR

The sensors found inside a consumer headset fall into a few functional groups. Inertial sensors measure the device's own motion. Optical and image sensors observe the outside world or the user's eyes. Depth sensors measure distance to surfaces. Environmental and presence sensors handle ambient conditions and wear detection.

Sensor	Measures	Typical VR/AR role
Accelerometer	Linear acceleration including gravity	Tilt reference and short-term motion in the IMU
Gyroscope	Angular velocity (rate of rotation)	Fast head rotation tracking (pitch, yaw, roll)
Magnetometer	Local magnetic field direction	Absolute heading reference to correct gyroscope drift
RGB camera	Visible-light images	Mixed-reality passthrough and color capture
Monochrome / IR camera	Visible or infrared images	Inside-out optical SLAM and controller LED tracking
Depth sensor	Distance to surfaces	Room mapping and 3D mesh generation
Eye-tracking camera	Pupil and corneal reflection	Gaze estimation and foveated rendering
Photodiode	Timed light pulses	Pose computation in laser base-station systems
Proximity sensor	Presence of the wearer	Automatic display and power on/off

Inertial sensors and the IMU

The single most important sensor cluster in a VR headset is the inertial measurement unit (IMU), a micro-electromechanical (MEMS) device that typically combines three sensors: a gyroscope, an accelerometer and a magnetometer. The gyroscope measures angular velocity around each axis, which gives fast and low-latency rotation tracking. The accelerometer measures proper acceleration, including the constant pull of gravity, so when the device is nearly still it can determine tilt relative to the vertical. The magnetometer acts as a digital compass, sensing the Earth's magnetic field to supply an absolute heading and to correct the slow drift that accumulates when gyroscope readings are integrated over time.

An IMU on its own cannot hold an accurate position. Integrating acceleration once gives velocity and integrating again gives position, but small errors compound quickly, so inertial data alone drifts within seconds. For this reason headsets that track position in six degrees of freedom fuse the IMU output with an optical source (cameras or base stations) using a sensor fusion algorithm, commonly a variant of the Kalman filter. The Kalman filter was introduced by Rudolf E. Kalman in 1960 as a method for estimating the state of a system from noisy measurements, and it remains the standard tool for combining inertial and optical sensor data in tracking systems.

Optical and image sensors

Cameras are the optical sensors that let a headset see the room and track objects in it. In an inside-out system the cameras are mounted on the headset and look outward, running optical SLAM on natural features of the surroundings to estimate the headset's own pose, and observing infrared light from the controllers to track them. In an outside-in system the cameras or photosensors observe markers, and the older Oculus Rift CV1 (2016) used an external infrared camera (the Oculus Sensor) to watch a pattern of infrared LEDs embedded under the surface of the headset and the Oculus Touch controllers, a system Oculus called Constellation.

Some tracking systems use a simpler optical sensor, the photodiode, rather than a full camera. In Valve's Lighthouse system, used by the HTC Vive, stationary base stations sweep the room with timed infrared laser planes, alternating horizontal and vertical, while arrays of photodiodes on the headset and controllers detect each sweep. From the precise timing of when each photodiode is hit, an onboard chip computes the device's pose. The cycle is 8.333 ms long, which corresponds to 120 Hz, and the original Vive controller carried 24 photodiode sensors around its ring.

Depth sensors

A depth sensor measures the distance from the headset to surfaces in the room, producing a depth map the system can turn into a 3D mesh for placing virtual objects and for guardian or boundary features. Two approaches dominate in VR and AR. A structured-light sensor projects a known infrared pattern, often thousands of dots, onto the scene; closer dots appear larger and farther dots appear smaller, so the deformation of the pattern is read by triangulation to recover depth. A time-of-flight (ToF) sensor instead measures the round-trip time of emitted light reflected back from objects; ToF reaches longer ranges but generally resolves less fine detail than structured light, and it is common for environment mapping and object detection. A scanning LiDAR is a related ranging sensor used for the same purpose at headset scale.

The Meta Quest 3 (2023) places an infrared patterned-light projector in its center module, marketed as a depth sensor, and uses it together with the headset's infrared cameras to read the depth and distance of the user's surroundings. The Apple Vision Pro (2024) includes a dedicated LiDAR Scanner and a TrueDepth camera alongside its main cameras for measuring the 3D structure of the environment.

Eye-tracking sensors

Eye tracking uses infrared illuminators and small infrared cameras pointed at the eyes from inside the headset. The illuminators cast invisible IR light onto the eye, which creates a reflection on the cornea (a glint); the cameras image the pupil and the glint, and software computes the gaze direction from the geometry between them, working in nearly any lighting because the illumination is supplied by the headset. Beyond interface input, gaze data drives foveated rendering, in which the system renders full detail only where the eye is looking and reduces detail in the periphery to lower the load on the GPU. The Apple Vision Pro uses four eye-tracking cameras for this purpose.

Environmental and presence sensors

Headsets also carry sensors that are not part of pose tracking. An ambient light sensor and a flicker sensor, both listed in the Apple Vision Pro specification, let the device adapt to room lighting. A proximity sensor detects when the user has put the headset on or taken it off: when it senses the wearer's presence the display turns on, and when the headset is removed the display turns off to save power. Such wear detection can use a capacitive sensor, because a human head produces a large capacitance change when it comes near the sensor, and some designs combine the proximity reading with the IMU to confirm a "donned" event.

Sensor arrays in current headsets

Modern standalone headsets carry more than a dozen sensors. The Meta Quest 3 front module holds two 4-megapixel RGB cameras for passthrough and four infrared cameras (two on the front and two on the underside) for inside-out tracking, plus the central depth projector and an IMU. The Apple Vision Pro sensor count is larger, as listed in Apple's own specification.

Apple Vision Pro sensor	Quantity
Main cameras	2
World-facing tracking cameras	6
Eye-tracking cameras	4
TrueDepth camera	1
LiDAR Scanner	1
Inertial measurement units (IMUs)	4
Flicker sensor	1
Ambient light sensor	1

Source: Apple Vision Pro technical specifications.

History in motion tracking

The use of inertial and optical sensors to track a head in space predates consumer VR. The mathematical basis for combining noisy sensor readings into a stable estimate, the Kalman filter, dates to 1960 and was first applied in aerospace navigation before becoming standard in robotics and tracking. The shift that made low-cost VR possible was the arrival of cheap MEMS inertial sensors in smartphones, which put accelerometers, gyroscopes and magnetometers into mass production. Early consumer headsets such as the Gear VR relied chiefly on these inertial sensors for 3-degree-of-freedom rotation, while later positional systems added optical sensors: external infrared cameras and LED markers in the Oculus Rift CV1 Constellation system and laser base stations with photodiodes in the HTC Vive Lighthouse system, both in 2016. From the Oculus Quest (2019) onward the industry moved to inside-out tracking, in which the headset's own cameras and IMU do all the work and no external sensors are required.