Passthrough: Difference between revisions
Xinreality (talk | contribs) |
Xinreality (talk | contribs) No edit summary |
||
| Line 4: | Line 4: | ||
While primarily a feature of VR headsets aiming to add environmental awareness or MR capabilities, it functions as a form of [[Augmented Reality]] (AR), often termed "Video See-Through AR" (VST AR) or sometimes "pseudo-AR," as opposed to "[[Optical See-Through]] AR" (OST AR) systems which use transparent displays.<ref name="SkarbezVSTvsOST">[https://www.researchgate.net/publication/315722770_Revisiting_Milgram_and_Kishino%27s_Reality-Virtuality_Continuum Revisiting Milgram and Kishino's Reality-Virtuality Continuum] - Discusses the spectrum including Video See-Through.</ref> Passthrough is a key enabler of [[mixed reality]] and [[spatial computing]] experiences on modern headsets. | While primarily a feature of VR headsets aiming to add environmental awareness or MR capabilities, it functions as a form of [[Augmented Reality]] (AR), often termed "Video See-Through AR" (VST AR) or sometimes "pseudo-AR," as opposed to "[[Optical See-Through]] AR" (OST AR) systems which use transparent displays.<ref name="SkarbezVSTvsOST">[https://www.researchgate.net/publication/315722770_Revisiting_Milgram_and_Kishino%27s_Reality-Virtuality_Continuum Revisiting Milgram and Kishino's Reality-Virtuality Continuum] - Discusses the spectrum including Video See-Through.</ref> Passthrough is a key enabler of [[mixed reality]] and [[spatial computing]] experiences on modern headsets. | ||
== Core Technology and How It Works == | ==Core Technology and How It Works== | ||
The fundamental principle of passthrough involves a real-time processing pipeline: | The fundamental principle of passthrough involves a real-time processing pipeline: | ||
| Line 15: | Line 15: | ||
# '''Display:''' The processed video feed is rendered onto the headset's internal [[display|displays]], replacing or being overlaid upon the virtual content. The primary goal is to achieve this entire pipeline with minimal [[latency (engineering)|latency]] (ideally under 20 milliseconds<ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref>) to avoid discomfort and maintain realism. | # '''Display:''' The processed video feed is rendered onto the headset's internal [[display|displays]], replacing or being overlaid upon the virtual content. The primary goal is to achieve this entire pipeline with minimal [[latency (engineering)|latency]] (ideally under 20 milliseconds<ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref>) to avoid discomfort and maintain realism. | ||
== History and Evolution == | ==History and Evolution== | ||
While the concept of video passthrough existed in research labs for decades,<ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> its implementation in consumer VR headsets evolved significantly: | While the concept of video passthrough existed in research labs for decades,<ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> its implementation in consumer VR headsets evolved significantly: | ||
| Line 32: | Line 32: | ||
Passthrough has evolved from a basic safety utility to a core feature enabling sophisticated mixed reality experiences, blurring the lines between traditional VR and AR. | Passthrough has evolved from a basic safety utility to a core feature enabling sophisticated mixed reality experiences, blurring the lines between traditional VR and AR. | ||
== Types of Passthrough == | ==Types of Passthrough== | ||
Passthrough implementations vary significantly. Key characteristics include: | Passthrough implementations vary significantly. Key characteristics include: | ||
=== Monochrome Passthrough === | ===Monochrome Passthrough=== | ||
Uses black-and-white camera feeds. Common in earlier VR headsets (Oculus Rift S, Quest 1 & 2) or as a design choice (PSVR2), often leveraging existing grayscale tracking cameras.<ref name="UploadVR_Q3Review_MR">UploadVR – Quest 3 Review (mixed reality section)</ref><ref name="RoadToVR_PSVR2"/> Provides basic environmental awareness but lacks color cues and realism. Advantages include potentially better low-light sensitivity and lower processing requirements.<ref name="PCMag_passthrough"/> | Uses black-and-white camera feeds. Common in earlier VR headsets (Oculus Rift S, Quest 1 & 2) or as a design choice (PSVR2), often leveraging existing grayscale tracking cameras.<ref name="UploadVR_Q3Review_MR">UploadVR – Quest 3 Review (mixed reality section)</ref><ref name="RoadToVR_PSVR2"/> Provides basic environmental awareness but lacks color cues and realism. Advantages include potentially better low-light sensitivity and lower processing requirements.<ref name="PCMag_passthrough"/> | ||
=== Color Passthrough === | ===Color Passthrough=== | ||
Uses [[RGB]] color cameras for a full-color view of the real world, greatly enhancing realism and enabling use cases like reading phone screens or interacting with colored objects. First widely available consumer example was Meta Quest Pro.<ref name="MixedNews_Cambria"/> Quality varies significantly based on camera resolution, processing, and calibration (e.g., Quest 3 offers ~10x the passthrough pixels of Quest 2).<ref name="UploadVR_specs">UploadVR – Quest 3 Specs Compared To Quest 2 & Apple Vision Pro (David Heaney, Sep 27, 2023)</ref> High-quality color passthrough (e.g., Varjo XR series, Vision Pro) aims for near-photorealism.<ref name="Skarredghost_Varjo"/><ref name="VisionProPassthrough"/> Requires more powerful hardware and sophisticated software. | Uses [[RGB]] color cameras for a full-color view of the real world, greatly enhancing realism and enabling use cases like reading phone screens or interacting with colored objects. First widely available consumer example was Meta Quest Pro.<ref name="MixedNews_Cambria"/> Quality varies significantly based on camera resolution, processing, and calibration (e.g., Quest 3 offers ~10x the passthrough pixels of Quest 2).<ref name="UploadVR_specs">UploadVR – Quest 3 Specs Compared To Quest 2 & Apple Vision Pro (David Heaney, Sep 27, 2023)</ref> High-quality color passthrough (e.g., Varjo XR series, Vision Pro) aims for near-photorealism.<ref name="Skarredghost_Varjo"/><ref name="VisionProPassthrough"/> Requires more powerful hardware and sophisticated software. | ||
=== Monoscopic vs. Stereoscopic === | ===Monoscopic vs. Stereoscopic=== | ||
*'''Monoscopic (2D):''' Uses a single camera view (or identical views) for both eyes (e.g., original HTC Vive, initial Pico 4 implementation<ref name="Reddit_PicoMono"/>). Lacks [[binocular disparity]], resulting in a "flat" image without true depth perception. Scale and distance can feel incorrect or uncomfortable. | *'''Monoscopic (2D):''' Uses a single camera view (or identical views) for both eyes (e.g., original HTC Vive, initial Pico 4 implementation<ref name="Reddit_PicoMono"/>). Lacks [[binocular disparity]], resulting in a "flat" image without true depth perception. Scale and distance can feel incorrect or uncomfortable. | ||
*'''Stereoscopic (3D):''' Uses two distinct camera viewpoints (one per eye, or reconstructed dual views) to create a 3D effect with depth perception. Requires cameras positioned roughly at the user's [[interpupillary distance]] (IPD) and careful calibration/reprojection. Essential for comfortable MR and accurate spatial interaction. Implemented in Rift S, PSVR2 (B&W stereo), Quest Pro, Quest 3, Vision Pro, Varjo XR series, etc.<ref name="UploadVR_Q3Review"/> Achieving correct scale and geometry is key to avoiding discomfort.<ref name="UploadVR_Q3Review_MR"/> | *'''Stereoscopic (3D):''' Uses two distinct camera viewpoints (one per eye, or reconstructed dual views) to create a 3D effect with depth perception. Requires cameras positioned roughly at the user's [[interpupillary distance]] (IPD) and careful calibration/reprojection. Essential for comfortable MR and accurate spatial interaction. Implemented in Rift S, PSVR2 (B&W stereo), Quest Pro, Quest 3, Vision Pro, Varjo XR series, etc.<ref name="UploadVR_Q3Review"/> Achieving correct scale and geometry is key to avoiding discomfort.<ref name="UploadVR_Q3Review_MR"/> | ||
=== Depth-Aware Passthrough === | ===Depth-Aware Passthrough=== | ||
Systems that actively measure or infer the distance to real-world objects and surfaces, integrating this [[depth map]] into the passthrough experience. This enables: | Systems that actively measure or infer the distance to real-world objects and surfaces, integrating this [[depth map]] into the passthrough experience. This enables: | ||
*'''Accurate placement and scaling''' of virtual objects relative to the real world. | *'''Accurate placement and scaling''' of virtual objects relative to the real world. | ||
| Line 55: | Line 55: | ||
Depth-aware passthrough significantly enhances MR realism and comfort, enabling features like automatic room scanning and persistent virtual object anchoring.<ref name="UploadVR_Q3Review_MR"/> | Depth-aware passthrough significantly enhances MR realism and comfort, enabling features like automatic room scanning and persistent virtual object anchoring.<ref name="UploadVR_Q3Review_MR"/> | ||
=== Mixed Reality Blending === | ===Mixed Reality Blending=== | ||
Refers to how seamlessly the passthrough system integrates virtual content with the real-world camera feed. Advanced implementations aim to unify lighting, shadows, reflections, and occlusion across both realities. Examples include: | Refers to how seamlessly the passthrough system integrates virtual content with the real-world camera feed. Advanced implementations aim to unify lighting, shadows, reflections, and occlusion across both realities. Examples include: | ||
*Virtual objects casting realistic shadows on real surfaces.<ref name="Varjo_blog">Varjo Blog – Video Pass-Through XR – Merge Real and Virtual (Urho Konttori, 2020)</ref> | *Virtual objects casting realistic shadows on real surfaces.<ref name="Varjo_blog">Varjo Blog – Video Pass-Through XR – Merge Real and Virtual (Urho Konttori, 2020)</ref> | ||
| Line 63: | Line 63: | ||
Requires high-quality color, stereoscopic depth, active depth sensing, low latency, and sophisticated rendering techniques (e.g., real-time lighting estimation, environmental mapping). Devices like Quest 3 and Vision Pro heavily emphasize these capabilities.<ref name="UploadVR_Q3Review_MR"/><ref name="Verge_VisionPro"/> | Requires high-quality color, stereoscopic depth, active depth sensing, low latency, and sophisticated rendering techniques (e.g., real-time lighting estimation, environmental mapping). Devices like Quest 3 and Vision Pro heavily emphasize these capabilities.<ref name="UploadVR_Q3Review_MR"/><ref name="Verge_VisionPro"/> | ||
== Technical Challenges == | ==Technical Challenges== | ||
Creating high-quality, comfortable passthrough involves overcoming significant hurdles: | Creating high-quality, comfortable passthrough involves overcoming significant hurdles: | ||
| Line 73: | Line 73: | ||
*'''[[Field of View]] (FOV):''' Passthrough FOV is often narrower than human vision or even the headset's display FOV, creating a "tunnel vision" effect or visible borders where the passthrough image ends. Wide-angle lenses used to increase FOV introduce distortion that needs correction. | *'''[[Field of View]] (FOV):''' Passthrough FOV is often narrower than human vision or even the headset's display FOV, creating a "tunnel vision" effect or visible borders where the passthrough image ends. Wide-angle lenses used to increase FOV introduce distortion that needs correction. | ||
== Modern Solutions and Advancements == | ==Modern Solutions and Advancements== | ||
Engineers employ various techniques to address passthrough challenges: | Engineers employ various techniques to address passthrough challenges: | ||
| Line 87: | Line 87: | ||
*'''User Experience (UX) Improvements:''' Features like a dedicated passthrough toggle button (PSVR2<ref name="RoadToVR_PSVR2"/>), automatic passthrough activation when nearing boundaries (Quest Guardian<ref name="UploadVR_Q3Review_MR"/>), and boundaryless MR modes enhance usability and seamlessly blend real/virtual interactions. | *'''User Experience (UX) Improvements:''' Features like a dedicated passthrough toggle button (PSVR2<ref name="RoadToVR_PSVR2"/>), automatic passthrough activation when nearing boundaries (Quest Guardian<ref name="UploadVR_Q3Review_MR"/>), and boundaryless MR modes enhance usability and seamlessly blend real/virtual interactions. | ||
== Applications and Use Cases == | ==Applications and Use Cases== | ||
Passthrough enables diverse applications by allowing users to interact with the real world while immersed: | Passthrough enables diverse applications by allowing users to interact with the real world while immersed: | ||
=== Consumer Uses === | ===Consumer Uses=== | ||
*'''Safety and Convenience:''' Defining play boundaries ([[Guardian system]], [[Chaperone (virtual reality)]]), avoiding obstacles, checking phones, finding controllers, or interacting briefly with people/pets without removing the headset.<ref name="PCMag_passthrough"/> | *'''Safety and Convenience:''' Defining play boundaries ([[Guardian system]], [[Chaperone (virtual reality)]]), avoiding obstacles, checking phones, finding controllers, or interacting briefly with people/pets without removing the headset.<ref name="PCMag_passthrough"/> | ||
*'''[[Mixed Reality]] Gaming and Entertainment:''' Games where virtual elements interact with the user's physical room (e.g., characters hiding behind real furniture, virtual objects placed on real tables).<ref name="UploadVR_Q3Review_MR"/> Creative apps allowing virtual painting on real walls. | *'''[[Mixed Reality]] Gaming and Entertainment:''' Games where virtual elements interact with the user's physical room (e.g., characters hiding behind real furniture, virtual objects placed on real tables).<ref name="UploadVR_Q3Review_MR"/> Creative apps allowing virtual painting on real walls. | ||
| Line 96: | Line 96: | ||
*'''Social Presence:''' Reducing isolation during VR use by allowing users to see others in the same physical space. Enabling co-located MR experiences where multiple users interact with shared virtual content in the same room. | *'''Social Presence:''' Reducing isolation during VR use by allowing users to see others in the same physical space. Enabling co-located MR experiences where multiple users interact with shared virtual content in the same room. | ||
=== Enterprise and Professional Uses === | ===Enterprise and Professional Uses=== | ||
*'''Collaboration:''' Design reviews where virtual prototypes are viewed in a real meeting room alongside physical mockups or colleagues.<ref name="XRToday_enterprise">XR Today – VR Passthrough in Enterprise (Immersive Learning News)</ref> Remote collaboration where experts guide on-site technicians using virtual annotations overlaid on the real equipment view. | *'''Collaboration:''' Design reviews where virtual prototypes are viewed in a real meeting room alongside physical mockups or colleagues.<ref name="XRToday_enterprise">XR Today – VR Passthrough in Enterprise (Immersive Learning News)</ref> Remote collaboration where experts guide on-site technicians using virtual annotations overlaid on the real equipment view. | ||
*'''Training and Simulation:''' Combining virtual scenarios with physical controls or environments (e.g., flight simulation using a real cockpit visible via passthrough, medical training on physical manikins with virtual overlays).<ref name="VIVE_Blog_Sauce"/> | *'''Training and Simulation:''' Combining virtual scenarios with physical controls or environments (e.g., flight simulation using a real cockpit visible via passthrough, medical training on physical manikins with virtual overlays).<ref name="VIVE_Blog_Sauce"/> | ||
| Line 102: | Line 102: | ||
*'''Productivity:''' Creating expansive virtual workspaces integrated with the physical office environment, improving multitasking while maintaining awareness.<ref name="XRToday_benefits">XR Today – What is VR Passthrough... (on benefits of passthrough)</ref> | *'''Productivity:''' Creating expansive virtual workspaces integrated with the physical office environment, improving multitasking while maintaining awareness.<ref name="XRToday_benefits">XR Today – What is VR Passthrough... (on benefits of passthrough)</ref> | ||
=== Industrial and Field Uses === | ===Industrial and Field Uses=== | ||
*'''Maintenance and Repair:''' Displaying step-by-step instructions, diagrams, or real-time data directly overlaid onto the machinery being worked on. | *'''Maintenance and Repair:''' Displaying step-by-step instructions, diagrams, or real-time data directly overlaid onto the machinery being worked on. | ||
*'''Assembly and Manufacturing:''' Providing guidance and quality control checks by highlighting parts or showing virtual indicators on physical products. | *'''Assembly and Manufacturing:''' Providing guidance and quality control checks by highlighting parts or showing virtual indicators on physical products. | ||
| Line 109: | Line 109: | ||
*'''Remote Operation:''' Controlling robots or drones using a passthrough view from the machine's perspective, augmented with virtual data displays. | *'''Remote Operation:''' Controlling robots or drones using a passthrough view from the machine's perspective, augmented with virtual data displays. | ||
== Comparison with Optical See-Through AR == | ==Comparison with Optical See-Through AR== | ||
Passthrough (Video See-Through, VST) is distinct from [[Optical See-Through]] (OST) AR, used by devices like [[Microsoft HoloLens|HoloLens]] and [[Magic Leap]]. | Passthrough (Video See-Through, VST) is distinct from [[Optical See-Through]] (OST) AR, used by devices like [[Microsoft HoloLens|HoloLens]] and [[Magic Leap]]. | ||
| Line 123: | Line 123: | ||
VST AR is currently favored in the consumer MR space, leveraging existing VR display technology, while OST AR maintains advantages for applications where unobstructed real-world vision is paramount. | VST AR is currently favored in the consumer MR space, leveraging existing VR display technology, while OST AR maintains advantages for applications where unobstructed real-world vision is paramount. | ||
== Notable Implementations == | ==Notable Implementations== | ||
*'''[[Meta Quest]] Series (Quest, Quest 2, Quest Pro, Quest 3):''' Evolved from basic monochrome safety features to sophisticated, depth-aware color passthrough using [[Machine Learning|ML]] reconstruction, making MR central to the platform.<ref name="Quest3PassthroughReview"/><ref name="QuestProPassthrough"/> | *'''[[Meta Quest]] Series (Quest, Quest 2, Quest Pro, Quest 3):''' Evolved from basic monochrome safety features to sophisticated, depth-aware color passthrough using [[Machine Learning|ML]] reconstruction, making MR central to the platform.<ref name="Quest3PassthroughReview"/><ref name="QuestProPassthrough"/> | ||
*'''[[Apple Vision Pro]]:''' High-resolution color passthrough as the default mode for "[[Spatial Computing]]", emphasizing low latency via a dedicated [[Apple R1]] chip and [[LiDAR]] for depth.<ref name="VisionProPassthrough"/> | *'''[[Apple Vision Pro]]:''' High-resolution color passthrough as the default mode for "[[Spatial Computing]]", emphasizing low latency via a dedicated [[Apple R1]] chip and [[LiDAR]] for depth.<ref name="VisionProPassthrough"/> | ||
| Line 133: | Line 133: | ||
*'''[[Valve Index]]:''' Basic stereoscopic monochrome passthrough via front cameras.<ref name="Index_Docs">Valve Index Hardware Documentation. (2019).</ref> | *'''[[Valve Index]]:''' Basic stereoscopic monochrome passthrough via front cameras.<ref name="Index_Docs">Valve Index Hardware Documentation. (2019).</ref> | ||
== Future Developments == | ==Future Developments== | ||
Ongoing research and development aim to improve passthrough by: | Ongoing research and development aim to improve passthrough by: | ||
* | *Achieving even lower latency and higher resolution/FOV, approaching the fidelity of human vision. | ||
* | *Improving camera [[dynamic range]], color fidelity, and low-light performance. | ||
* | *Developing more sophisticated and efficient [[depth sensing]] and real-time 3D reconstruction (e.g., using [[LiDAR]], advanced [[Computer Vision|CV]], [[Neural Radiance Fields|NeRFs]]). | ||
* | *Integrating [[Artificial Intelligence|AI]] for enhanced scene understanding, object recognition, segmentation, and interaction modeling (realistic physics, occlusion). | ||
* | *Implementing selective passthrough (showing only specific real-world elements like hands or keyboards) and potentially "augmented reality" filters applied to the real-world view. | ||
* | *Utilizing [[eye tracking]] for [[foveated rendering]] of the passthrough feed or dynamic depth-of-field adjustments. | ||
* | *Exploring novel camera technologies like light field cameras (e.g., Meta's "Flamera" concept<ref name="KGuttag_Flamera">KGOnTech (Karl Guttag) – Meta Flamera Light Field Passthrough</ref>) to better solve perspective issues. | ||
As technology matures, VST passthrough aims to provide a near-seamless blend between the virtual and physical worlds, potentially unifying VR and AR capabilities into single, versatile devices. | As technology matures, VST passthrough aims to provide a near-seamless blend between the virtual and physical worlds, potentially unifying VR and AR capabilities into single, versatile devices. | ||
== See Also == | ==See Also== | ||
* [[Mixed Reality]] (MR) | * [[Mixed Reality]] (MR) | ||
* [[Augmented Reality]] (AR) | * [[Augmented Reality]] (AR) | ||
| Line 165: | Line 165: | ||
* [[Apple Vision Pro]] | * [[Apple Vision Pro]] | ||
== References == | ==References== | ||
<references> | <references> | ||
<ref name="XRToday_def">XR Today – What is VR Passthrough and How is it Shaping the Future of XR? (Immersive Learning News, Dec 2024)</ref> | <ref name="XRToday_def">XR Today – What is VR Passthrough and How is it Shaping the Future of XR? (Immersive Learning News, Dec 2024)</ref> | ||