Passthrough

See also: Terms and Technical Terms

Passthrough, often referred to as video passthrough, is a feature found in Virtual Reality (VR) and Mixed Reality (MR) headsets that utilizes external cameras to capture a live video feed of the physical environment around the user and display it on the internal screens within the headset. This capability effectively allows users to see the "real world" without removing the headset, bridging the gap between fully immersive virtual experiences and the user's actual surroundings.

While primarily a feature of VR headsets aiming to add environmental awareness or MR capabilities, it functions as a form of Augmented Reality (AR), often termed "Video See-Through AR" (VST AR) or sometimes "pseudo-AR," as opposed to "Optical See-Through AR" (OST AR) systems which use transparent displays. Passthrough is a key enabler of mixed reality and spatial computing experiences on modern headsets.

Core Technology and How It Works

The fundamental principle of passthrough involves a real-time processing pipeline:

1. Capture: One or more outward-facing digital cameras mounted on the headset capture video of the external world. Early or basic systems might use a single camera (providing a monoscopic view), while more advanced systems use two or more cameras to capture stereoscopic video, enabling depth perception. Modern systems often use a combination of RGB color cameras and monochrome (grayscale) sensors for different purposes (for example capturing color data vs. motion/detail).
2. Processing: The captured video footage is sent to the headset's processor (either an onboard SoC or a connected PC's GPU). This stage is computationally intensive and critical for a usable and comfortable experience. It typically involves several steps:
   • Rectification/Undistortion: Correcting lens distortion inherent in the wide-angle cameras typically used to maximize FOV.
   • Reprojection/Warping: Adjusting the captured image perspective to align with the user's eye position inside the headset, rather than the camera's physical position on the outside. This difference in viewpoint causes parallax, and correcting it ("perspective correction") is crucial for accurate spatial representation, correct scale perception, and minimizing motion sickness. Algorithms based on Computer Vision and potentially IMU sensor data are used. Some modern headsets, like the Meta Quest Pro and Meta Quest 3, employ Machine Learning or neural networks to improve the realism and accuracy of this reconstruction.
   • Sensor Fusion: Combining data from multiple cameras (for example fusing monochrome detail with RGB color) and integrating tracking data (for example from inside-out tracking sensors or depth sensors) to ensure the passthrough view remains stable, depth-correct, and aligned with the user's head movements.
   • Color Correction & Enhancement: Adjusting colors, brightness, and contrast to appear more natural, especially under varying lighting conditions. This can also involve AI-based denoising or upscaling.
3. Display: The processed video feed is rendered onto the headset's internal displays, replacing or being overlaid upon the virtual content. The primary goal is to achieve this entire pipeline with minimal latency (ideally under 20 milliseconds) to avoid discomfort and maintain realism.

History and Evolution

While the concept of video passthrough existed in research labs for decades, its implementation in consumer VR headsets evolved significantly:

• Early Stages (Mid-2010s): Passthrough began appearing primarily as a safety feature. In 2016, the HTC Vive prototype (Vive Pre) introduced a front-facing camera providing a basic, monochrome, 2D view for obstacle avoidance. Valve's software projected this onto a virtual sphere to approximate perspective. It was low-resolution and intended for brief checks.
• Integrated Monochrome (Late 2010s): Headsets using inside-out tracking leveraged their tracking cameras for improved passthrough. The Oculus Rift S (2019) offered "Passthrough+" using its multiple monochrome cameras for a stereoscopic view. The original Oculus Quest (2019) and Meta Quest 2 (2020) provided similar basic monochrome passthrough, mainly for setting up the Guardian system and quick environment checks.
• Early Mixed Reality Steps (Early 2020s): In 2021, Meta released an experimental Passthrough API for Quest 2 developers, allowing apps to overlay virtual elements onto the monochrome feed, marking a step towards consumer MR. Simultaneously, enterprise headsets like the Varjo XR-1 (2019) and XR-3 (2021) pushed high-fidelity color passthrough with dual high-resolution cameras, setting a benchmark for quality.
• Mainstream Color Passthrough (2022-Present):
  • The Meta Quest Pro (2022) was the first major consumer headset featuring high-quality, stereoscopic color passthrough, using a novel camera array (monochrome for depth/detail, RGB for color) and ML reconstruction.
  • Competitors like the Pico 4 (late 2022) and HTC Vive XR Elite (2023) also introduced color passthrough, although early implementations like the Pico 4's were initially monoscopic and lacked depth correction.
  • Sony's PlayStation VR2 (2023) included stereo passthrough, but kept it black-and-white, accessible via a dedicated button for quick checks.
  • The Meta Quest 3 (late 2023) brought high-resolution stereo color passthrough with an active depth sensor (structured light projector) to the mainstream consumer market, offering significantly improved clarity and depth accuracy over Quest 2 and Quest Pro.
  • The Apple Vision Pro (2023 announcement, 2024 release) emphasized passthrough-based MR ("spatial computing"), using dual high-resolution color cameras, advanced processing (Apple R1 chip), and a LiDAR scanner for precise depth mapping.
  • Other high-end devices like the Pimax Crystal (2023) and Varjo XR-4 (late 2023) continued to push resolution and fidelity.
  • Even mid-range devices began incorporating improved color passthrough and depth sensing, like the anticipated Pico 4 Ultra (2024).

Passthrough has evolved from a basic safety utility to a core feature enabling sophisticated mixed reality experiences, blurring the lines between traditional VR and AR.

Types of Passthrough

Passthrough implementations vary significantly. Key characteristics include:

Monochrome Passthrough

Uses black-and-white camera feeds. Common in earlier VR headsets (Oculus Rift S, Quest 1 & 2) or as a design choice (PSVR2), often leveraging existing grayscale tracking cameras. Provides basic environmental awareness but lacks color cues and realism. Advantages include potentially better low-light sensitivity and lower processing requirements.

Color Passthrough

Uses RGB color cameras for a full-color view of the real world, greatly enhancing realism and enabling use cases like reading phone screens or interacting with colored objects. First widely available consumer example was Meta Quest Pro. Quality varies significantly based on camera resolution, processing, and calibration (for example Quest 3 offers ~10x the passthrough pixels of Quest 2). High-quality color passthrough (for example Varjo XR series, Vision Pro) aims for near-photorealism. Requires more powerful hardware and sophisticated software.

Monoscopic vs. Stereoscopic

• Monoscopic (2D): Uses a single camera view (or identical views) for both eyes (for example original HTC Vive, initial Pico 4 implementation). Lacks binocular disparity, resulting in a "flat" image without true depth perception. Scale and distance can feel incorrect or uncomfortable.
• Stereoscopic (3D): Uses two distinct camera viewpoints (one per eye, or reconstructed dual views) to create a 3D effect with depth perception. Requires cameras positioned roughly at the user's interpupillary distance (IPD) and careful calibration/reprojection. Essential for comfortable MR and accurate spatial interaction. Implemented in Rift S, PSVR2 (B&W stereo), Quest Pro, Quest 3, Vision Pro, Varjo XR series, etc. Achieving correct scale and geometry is key to avoiding discomfort.

Depth-Aware Passthrough

Systems that actively measure or infer the distance to real-world objects and surfaces, integrating this depth map into the passthrough experience. This enables:

• Accurate placement and scaling of virtual objects relative to the real world.
• Occlusion: Allowing virtual objects to realistically appear behind real objects (and vice-versa).
• Improved interaction: Understanding the geometry of the environment for physics and hand interactions.

Methods include:

• Passive Stereo Vision: Calculating depth from the differences between two camera images (computationally intensive, can struggle with textureless surfaces).
• Active Depth Sensing: Using dedicated sensors like Infrared (IR) projectors (Quest Pro inference), Structured Light projectors (Quest 3), Time-of-Flight (ToF) sensors, or LiDAR (Vision Pro). These provide more robust and direct depth measurements.

Depth-aware passthrough significantly enhances MR realism and comfort, enabling features like automatic room scanning and persistent virtual object anchoring.

Mixed Reality Blending

Refers to how seamlessly the passthrough system integrates virtual content with the real-world camera feed. Advanced implementations aim to unify lighting, shadows, reflections, and occlusion across both realities. Examples include:

• Virtual objects casting realistic shadows on real surfaces.
• Virtual elements being correctly hidden by real furniture or people.
• Virtual lighting affecting the appearance of the real world within the passthrough view (and vice-versa).
• Using ML for scene segmentation (identifying walls, floors, furniture, people) to enable complex interactions.

Requires high-quality color, stereoscopic depth, active depth sensing, low latency, and sophisticated rendering techniques (for example real-time lighting estimation, environmental mapping). Devices like Quest 3 and Vision Pro heavily emphasize these capabilities.

Technical Challenges

Creating high-quality, comfortable passthrough involves overcoming significant hurdles:

• Latency: The delay between real-world motion and the passthrough display update (photon-to-photon latency). High latency (>~20ms) causes disorientation, motion sickness ("world swimming"), and breaks immersion. Fast processing pipelines are essential. Residual latency can cause ghosting or trailing artifacts on moving objects.
• Resolution and Image Quality: Camera feeds are often lower resolution than human vision, leading to pixelation or blurriness, making fine details (like text) hard to see. Limited dynamic range struggles with bright highlights and dark shadows compared to the human eye. Poor low-light performance results in noisy, grainy images. Achieving high resolution and good image quality requires better sensors and significant processing power.
• Camera Placement and Perspective Mismatch: Cameras are offset from the user's eyes, causing parallax errors if not corrected. Naive display leads to distorted views, incorrect scale, and depth perception issues, especially for close objects. Sophisticated reprojection algorithms are needed to warp the camera view to match the eye's perspective, but perfect correction is difficult. This geometric misalignment can cause eye strain or discomfort. Close objects (<~0.5m) often appear warped even in good systems due to sensor/lens limitations and reprojection challenges.
• Depth Perception and Occlusion: Even with stereo cameras, accurately replicating human depth perception is hard. Incorrect IPD matching or calibration can lead to scale issues. Lack of accurate, real-time depth maps makes correct occlusion (virtual behind real) difficult, breaking immersion. Errors in depth sensing or fusion can cause virtual objects to flicker or appear incorrectly positioned.
• Color Accuracy and Calibration: Matching the colors and brightness of the passthrough feed to both the real world and virtual elements is challenging. Poor white balance or color calibration makes the view look unnatural or filtered. Display limitations also affect color reproduction. Consistent calibration across cameras and over time (accounting for thermal drift) is crucial.
• Field of View (FOV): Passthrough FOV is often narrower than human vision or even the headset's display FOV, creating a "tunnel vision" effect or visible borders where the passthrough image ends. Wide-angle lenses used to increase FOV introduce distortion that needs correction.

Modern Solutions and Advancements

Engineers employ various techniques to address passthrough challenges:

• Multi-Camera Sensor Fusion: Using multiple cameras with different strengths (for example high-resolution RGB for color, fast monochrome for low-latency motion/detail) and fusing their data computationally. Overlapping camera views help compute stereo depth and increase effective FOV.
• Active Depth Sensing: Incorporating dedicated depth sensors (IR ToF, Structured Light, LiDAR) provides robust, real-time 3D geometry information of the environment, improving reprojection accuracy, occlusion handling, and spatial anchoring. This enables features like quick room meshing via APIs (for example Meta's Spatial Anchors, Apple's ARKit/RoomPlan).
• Machine Learning Enhancements: Using AI/ML for various tasks:
  • Image upscaling and denoising to improve clarity, especially in low light.
  • Advanced reprojection algorithms for more accurate perspective correction.
  • Scene segmentation to identify objects (hands, people, furniture) for better interaction and occlusion.
  • Improving SLAM for more stable tracking and anchoring of virtual objects.
• Reprojection and Virtual Cameras: Software techniques that warp the captured camera images based on depth data to synthesize a view from the user's actual eye positions ("virtual cameras"). Time-warping techniques can further reduce perceived latency by adjusting the image based on last-moment head movements.
• Improved Optics and Displays: Pancake lenses allow for thinner headsets where cameras can potentially be placed closer to the eyes, reducing offset. Higher resolution, higher dynamic range (for example Micro-OLED in Vision Pro), and faster refresh rate displays improve the fidelity of the displayed passthrough feed. Careful calibration of lens distortion profiles is also applied.
• User Experience (UX) Improvements: Features like a dedicated passthrough toggle button (PSVR2), automatic passthrough activation when nearing boundaries (Quest Guardian), and boundaryless MR modes enhance usability and seamlessly blend real/virtual interactions.

Applications and Use Cases

Passthrough enables diverse applications by allowing users to interact with the real world while immersed:

Consumer Uses

• Safety and Convenience: Defining play boundaries (Guardian system, Chaperone (virtual reality)), avoiding obstacles, checking phones, finding controllers, or interacting briefly with people/pets without removing the headset.
• Mixed Reality Gaming and Entertainment: Games where virtual elements interact with the user's physical room (for example characters hiding behind real furniture, virtual objects placed on real tables). Creative apps allowing virtual painting on real walls.
• Productivity and Utility: Using virtual desktops or multiple virtual monitors while still seeing the real keyboard, mouse, and desk.
• Social Presence: Reducing isolation during VR use by allowing users to see others in the same physical space. Enabling co-located MR experiences where multiple users interact with shared virtual content in the same room.

Enterprise and Professional Uses

• Collaboration: Design reviews where virtual prototypes are viewed in a real meeting room alongside physical mockups or colleagues. Remote collaboration where experts guide on-site technicians using virtual annotations overlaid on the real equipment view.
• Training and Simulation: Combining virtual scenarios with physical controls or environments (for example flight simulation using a real cockpit visible via passthrough, medical training on physical manikins with virtual overlays).
• Visualization: Architects visualizing 3D models on a real site, designers overlaying virtual concepts onto physical products.
• Productivity: Creating expansive virtual workspaces integrated with the physical office environment, improving multitasking while maintaining awareness.

Industrial and Field Uses

• Maintenance and Repair: Displaying step-by-step instructions, diagrams, or real-time data directly overlaid onto the machinery being worked on.
• Assembly and Manufacturing: Providing guidance and quality control checks by highlighting parts or showing virtual indicators on physical products.
• Logistics: Warehouse workers seeing picking information or navigation paths overlaid onto the real warehouse environment.
• Construction: On-site visualization of BIM models overlaid onto the actual construction progress for inspection and alignment checks.
• Remote Operation: Controlling robots or drones using a passthrough view from the machine's perspective, augmented with virtual data displays.

Comparison with Optical See-Through AR

Passthrough (Video See-Through, VST) is distinct from Optical See-Through (OST) AR, used by devices like HoloLens and Magic Leap.

Optical See-Through (OST)

• Uses semi-transparent displays (waveguides, birdbath optics) allowing direct view of the real world. Virtual images are projected onto these combiners.
• Pros: Real world is seen perfectly naturally (zero latency, full resolution/color/dynamic range). Lower power consumption for viewing the real world. Solid real-world objects are always solid.
• Cons: Virtual elements often appear transparent or "ghostly," lacking solidity. Limited FOV for virtual content is common. Difficulty displaying black (virtual content is additive). Virtual content can be washed out by bright ambient light. Accurate alignment ("registration") of virtual content to the real world can be challenging. Cannot computationally modify the real-world view.

Video Passthrough (VST)

• Uses cameras and opaque displays to show a reconstructed view of the real world.
• Pros: Virtual elements can be fully opaque and seamlessly blended. Potential for wider FOV matching the VR display. Can computationally modify the real-world view (for example brightness enhancement, selective filtering). Better blocking of ambient light for virtual content.
• Cons: Real-world view is mediated by technology, subject to limitations (latency, resolution, color, dynamic range, distortion). Higher power consumption. Potential for discomfort (motion sickness, eye strain) if not implemented well. Real-world objects might appear less "solid" due to latency or artifacts.

VST AR is currently favored in the consumer MR space, leveraging existing VR display technology, while OST AR maintains advantages for applications where unobstructed real-world vision is paramount.

Notable Implementations

• Meta Quest Series (Quest, Quest 2, Quest Pro, Quest 3): Evolved from basic monochrome safety features to sophisticated, depth-aware color passthrough using ML reconstruction, making MR central to the platform.
• Apple Vision Pro: High-resolution color passthrough as the default mode for "Spatial Computing", emphasizing low latency via a dedicated Apple R1 chip and LiDAR for depth.
• Varjo XR Series (XR-1, XR-3, XR-4): Industry benchmark for high-fidelity, low-latency color passthrough, aimed at professional/enterprise markets.
• HTC Vive XR Elite: Offers color passthrough with a depth sensor for MR capabilities.
• Pimax Crystal: High-resolution VR headset incorporating color passthrough features.
• Lynx R1: Standalone headset project focusing specifically on delivering quality color passthrough at a competitive price point.
• PlayStation VR2: Features stereo black-and-white passthrough primarily for setup and quick environment checks.
• Valve Index: Basic stereoscopic monochrome passthrough via front cameras.

Future Developments

Ongoing research and development aim to improve passthrough by:

• Achieving even lower latency and higher resolution/FOV, approaching the fidelity of human vision.
• Improving camera dynamic range, color fidelity, and low-light performance.
• Developing more sophisticated and efficient depth sensing and real-time 3D reconstruction (for example using LiDAR, advanced CV, NeRFs).
• Integrating AI for enhanced scene understanding, object recognition, segmentation, and interaction modeling (realistic physics, occlusion).
• Implementing selective passthrough (showing only specific real-world elements like hands or keyboards) and potentially "augmented reality" filters applied to the real-world view.
• Utilizing eye tracking for foveated rendering of the passthrough feed or dynamic depth-of-field adjustments.
• Exploring novel camera technologies like light field cameras (for example Meta's "Flamera" concept) to better solve perspective issues.

As technology matures, VST passthrough aims to provide a near-seamless blend between the virtual and physical worlds, potentially unifying VR and AR capabilities into single, versatile devices.

See Also

• Mixed Reality (MR)
• Augmented Reality (AR)
• Virtual Reality (VR)
• Spatial computing
• Latency (engineering)
• Computer Vision
• Head-mounted display (HMD)
• Depth map
• Simultaneous localization and mapping (SLAM)
• Roomscale VR
• Guardian system (Meta)
• Chaperone (virtual reality) (SteamVR)
• Optical See-Through
• Microsoft HoloLens
• Magic Leap
• Varjo
• Meta Quest 3
• Apple Vision Pro