Passthrough: Difference between revisions
Xinreality (talk | contribs) No edit summary |
Xinreality (talk | contribs) No edit summary |
||
| Line 7: | Line 7: | ||
The fundamental principle of passthrough involves a real-time processing pipeline: | The fundamental principle of passthrough involves a real-time processing pipeline: | ||
# '''Capture:''' One or more outward-facing digital cameras mounted on the headset capture video of the external world. Early or basic systems might use a single camera (providing a monoscopic view), while more advanced systems use two or more cameras to capture [[stereoscopic]] video, enabling [[depth perception]].<ref name="StereoPassthrough">[https://ieeexplore.ieee.org/document/9191148 Example paper discussing stereoscopic passthrough challenges]</ref> Modern systems often use a combination of [[RGB]] color cameras and monochrome (grayscale) sensors for different purposes (for example capturing color data vs. motion/detail).<ref name="MixedNews_Cambria"> | # '''Capture:''' One or more outward-facing digital cameras mounted on the headset capture video of the external world. Early or basic systems might use a single camera (providing a monoscopic view), while more advanced systems use two or more cameras to capture [[stereoscopic]] video, enabling [[depth perception]].<ref name="StereoPassthrough">[https://ieeexplore.ieee.org/document/9191148 Example paper discussing stereoscopic passthrough challenges]</ref> Modern systems often use a combination of [[RGB]] color cameras and monochrome (grayscale) sensors for different purposes (for example capturing color data vs. motion/detail).<ref name="MixedNews_Cambria"></ref> | ||
# '''Processing:''' The captured video footage is sent to the headset's [[processor]] (either an onboard [[System on a Chip|SoC]] or a connected PC's [[GPU]]). This stage is computationally intensive and critical for a usable and comfortable experience. It typically involves several steps: | # '''Processing:''' The captured video footage is sent to the headset's [[processor]] (either an onboard [[System on a Chip|SoC]] or a connected PC's [[GPU]]). This stage is computationally intensive and critical for a usable and comfortable experience. It typically involves several steps: | ||
#* '''Rectification/Undistortion:''' Correcting [[lens distortion]] inherent in the wide-angle cameras typically used to maximize [[field of view|FOV]]. | #* '''Rectification/Undistortion:''' Correcting [[lens distortion]] inherent in the wide-angle cameras typically used to maximize [[field of view|FOV]]. | ||
#* '''Reprojection/Warping:''' Adjusting the captured image perspective to align with the user's eye position inside the headset, rather than the camera's physical position on the outside. This difference in viewpoint causes [[parallax]], and correcting it ("perspective correction") is crucial for accurate spatial representation, correct scale perception, and minimizing [[motion sickness]].<ref name="PassthroughChallengesUploadVR"> | #* '''Reprojection/Warping:''' Adjusting the captured image perspective to align with the user's eye position inside the headset, rather than the camera's physical position on the outside. This difference in viewpoint causes [[parallax]], and correcting it ("perspective correction") is crucial for accurate spatial representation, correct scale perception, and minimizing [[motion sickness]].<ref name="PassthroughChallengesUploadVR"></ref><ref name="KGuttag_Align"></ref> Algorithms based on [[Computer Vision]] and potentially [[Inertial Measurement Unit|IMU]] sensor data are used. Some modern headsets, like the [[Meta Quest Pro]] and [[Meta Quest 3]], employ [[Machine Learning]] or [[Neural Network|neural networks]] to improve the realism and accuracy of this reconstruction.<ref name="QuestProPassthrough">[https://www.meta.com/blog/quest/meta-reality-passthrough-quest-pro/ Meta Blog: Inside Meta Reality and Passthrough on Quest Pro]</ref> | ||
#* '''[[Sensor Fusion]]:''' Combining data from multiple cameras (for example fusing monochrome detail with RGB color<ref name="MixedNews_Cambria"/>) and integrating tracking data (for example from [[inside-out tracking]] sensors or [[depth sensor]]s) to ensure the passthrough view remains stable, depth-correct, and aligned with the user's head movements. | #* '''[[Sensor Fusion]]:''' Combining data from multiple cameras (for example fusing monochrome detail with RGB color<ref name="MixedNews_Cambria"/>) and integrating tracking data (for example from [[inside-out tracking]] sensors or [[depth sensor]]s) to ensure the passthrough view remains stable, depth-correct, and aligned with the user's head movements. | ||
#* '''Color Correction & Enhancement:''' Adjusting colors, brightness, and contrast to appear more natural, especially under varying lighting conditions. This can also involve [[Artificial Intelligence|AI]]-based denoising or upscaling.<ref name="UploadVR_Q3Review"> | #* '''Color Correction & Enhancement:''' Adjusting colors, brightness, and contrast to appear more natural, especially under varying lighting conditions. This can also involve [[Artificial Intelligence|AI]]-based denoising or upscaling.<ref name="UploadVR_Q3Review"></ref> | ||
# '''Display:''' The processed video feed is rendered onto the headset's internal [[display|displays]], replacing or being overlaid upon the virtual content. The primary goal is to achieve this entire pipeline with minimal [[latency (engineering)|latency]] (ideally under 20 milliseconds<ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref>) to avoid discomfort and maintain realism. | # '''Display:''' The processed video feed is rendered onto the headset's internal [[display|displays]], replacing or being overlaid upon the virtual content. The primary goal is to achieve this entire pipeline with minimal [[latency (engineering)|latency]] (ideally under 20 milliseconds<ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref>) to avoid discomfort and maintain realism. | ||
| Line 18: | Line 18: | ||
While the concept of video passthrough existed in research labs for decades,<ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> its implementation in consumer VR headsets evolved significantly: | While the concept of video passthrough existed in research labs for decades,<ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> its implementation in consumer VR headsets evolved significantly: | ||
*'''Early Stages (Mid-2010s):''' Passthrough began appearing primarily as a safety feature. In 2016, the [[HTC Vive]] prototype (Vive Pre) introduced a front-facing camera providing a basic, monochrome, 2D view for obstacle avoidance. Valve's software projected this onto a virtual sphere to approximate perspective.<ref name="RoadToVR_Vive"> | *'''Early Stages (Mid-2010s):''' Passthrough began appearing primarily as a safety feature. In 2016, the [[HTC Vive]] prototype (Vive Pre) introduced a front-facing camera providing a basic, monochrome, 2D view for obstacle avoidance. Valve's software projected this onto a virtual sphere to approximate perspective.<ref name="RoadToVR_Vive"></ref> It was low-resolution and intended for brief checks. | ||
*'''Integrated Monochrome (Late 2010s):''' Headsets using inside-out tracking leveraged their tracking cameras for improved passthrough. The [[Oculus Rift S]] (2019) offered "Passthrough+" using its multiple monochrome cameras for a stereoscopic view.<ref name="RiftS_Docs"> | *'''Integrated Monochrome (Late 2010s):''' Headsets using inside-out tracking leveraged their tracking cameras for improved passthrough. The [[Oculus Rift S]] (2019) offered "Passthrough+" using its multiple monochrome cameras for a stereoscopic view.<ref name="RiftS_Docs"></ref> The original [[Meta Quest|Oculus Quest]] (2019) and [[Meta Quest 2]] (2020) provided similar basic monochrome passthrough, mainly for setting up the [[Guardian system]] and quick environment checks.<ref name="Quest2Passthrough">[https://www.meta.com/blog/quest/oculus-quest-2-passthrough-public-api-passthrough/ Quest Blog on Quest 2 Passthrough improvements]</ref> | ||
*'''Early Mixed Reality Steps (Early 2020s):''' In 2021, Meta released an experimental Passthrough API for Quest 2 developers, allowing apps to overlay virtual elements onto the monochrome feed, marking a step towards consumer MR.<ref name="PCMag_passthrough">[https://developers.meta.com/horizon/blog/mixed-reality-with-passthrough/ Meta Developer Blog – Oculus Experiments With Mixed Reality via New Passthrough API] (July 25 2021)</ref> Simultaneously, enterprise headsets like the [[Varjo]] XR-1 (2019) and XR-3 (2021) pushed high-fidelity color passthrough with dual high-resolution cameras, setting a benchmark for quality.<ref name="Skarredghost_Varjo"> | *'''Early Mixed Reality Steps (Early 2020s):''' In 2021, Meta released an experimental Passthrough API for Quest 2 developers, allowing apps to overlay virtual elements onto the monochrome feed, marking a step towards consumer MR.<ref name="PCMag_passthrough">[https://developers.meta.com/horizon/blog/mixed-reality-with-passthrough/ Meta Developer Blog – Oculus Experiments With Mixed Reality via New Passthrough API] (July 25 2021)</ref> Simultaneously, enterprise headsets like the [[Varjo]] XR-1 (2019) and XR-3 (2021) pushed high-fidelity color passthrough with dual high-resolution cameras, setting a benchmark for quality.<ref name="Skarredghost_Varjo"></ref> | ||
*'''Mainstream Color Passthrough (2022-Present):''' | *'''Mainstream Color Passthrough (2022-Present):''' | ||
**The [[Meta Quest Pro]] (2022) was the first major consumer headset featuring high-quality, stereoscopic color passthrough, using a novel camera array (monochrome for depth/detail, RGB for color) and ML reconstruction.<ref name="MixedNews_Cambria"/> | **The [[Meta Quest Pro]] (2022) was the first major consumer headset featuring high-quality, stereoscopic color passthrough, using a novel camera array (monochrome for depth/detail, RGB for color) and ML reconstruction.<ref name="MixedNews_Cambria"/> | ||
| Line 171: | Line 171: | ||
<ref name="SkarbezVSTvsOST">[https://www.researchgate.net/publication/315722770_Revisiting_Milgram_and_Kishino%27s_Reality-Virtuality_Continuum Revisiting Milgram and Kishino's Reality-Virtuality Continuum] - Discusses the spectrum including Video See-Through.</ref> | <ref name="SkarbezVSTvsOST">[https://www.researchgate.net/publication/315722770_Revisiting_Milgram_and_Kishino%27s_Reality-Virtuality_Continuum Revisiting Milgram and Kishino's Reality-Virtuality Continuum] - Discusses the spectrum including Video See-Through.</ref> | ||
<ref name="StereoPassthrough">[https://ieeexplore.ieee.org/document/9191148 Example paper discussing stereoscopic passthrough challenges]</ref> | <ref name="StereoPassthrough">[https://ieeexplore.ieee.org/document/9191148 Example paper discussing stereoscopic passthrough challenges]</ref> | ||
<ref name="MixedNews_Cambria"> | <ref name="MixedNews_Cambria">[https://mixed-news.com/en/project-cambria-meta-explains-new-passthrough-technology/ MIXED News – Project Cambria: Meta explains new passthrough technology] (May 16 2022)</ref> | ||
<ref name="PassthroughChallengesUploadVR">[https://uploadvr.com/passthrough- | <ref name="PassthroughChallengesUploadVR">[https://www.uploadvr.com/quest-3-passthrough-will-improve-meta-cto/ UploadVR – Passthrough AR: The Technical Challenges of Blending Realities] (Oct 23 2023)</ref> | ||
<ref name="KGuttag_Align"> | <ref name="KGuttag_Align">[https://kguttag.com/2023/09/26/apple-vision-pro-part-6-passthrough-mixed-reality-ptmr-problems/ KGOnTech – Perspective‑Correct Passthrough] (Sept 26 2023)</ref> | ||
<ref name="QuestProPassthrough">[https://www.meta.com/blog/quest/meta-reality-passthrough-quest-pro/ Meta Blog: Inside Meta Reality and Passthrough on Quest Pro]</ref> | <ref name="QuestProPassthrough">[https://www.meta.com/blog/quest/meta-reality-passthrough-quest-pro/ Meta Blog: Inside Meta Reality and Passthrough on Quest Pro]</ref> | ||
<ref name="UploadVR_Q3Review">UploadVR - Quest 3 Review: Excellent VR With Limited MR (David Heaney, Oct 9, 2023)</ref> | <ref name="UploadVR_Q3Review">UploadVR - Quest 3 Review: Excellent VR With Limited MR (David Heaney, Oct 9, 2023)</ref> | ||
<ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref> | <ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref> | ||
<ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> | <ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> | ||
<ref name="RoadToVR_Vive"> | <ref name="RoadToVR_Vive">[https://www.roadtovr.com/htc-vive-front-facing-camera-video/ Road to VR – 8 Minutes of the HTC Vive’s Front‑facing Camera in Action] (Mar 10 2016)</ref> | ||
<ref name="RiftS_Docs"> | <ref name="RiftS_Docs">[https://www.meta.com/help/quest/459034058728993/ Oculus Rift S Product Documentation] (2019)</ref> | ||
<ref name="Quest2Passthrough">[https://www.meta.com/blog/quest/oculus-quest-2-passthrough-public-api-passthrough/ Quest Blog on Quest 2 Passthrough improvements]</ref> | <ref name="Quest2Passthrough">[https://www.meta.com/blog/quest/oculus-quest-2-passthrough-public-api-passthrough/ Quest Blog on Quest 2 Passthrough improvements]</ref> | ||
<ref name="PCMag_passthrough">PCMag - Oculus Experiments With Mixed Reality via New Passthrough API (Nathaniel Mott, July 25, 2021)</ref> | <ref name="PCMag_passthrough">PCMag - Oculus Experiments With Mixed Reality via New Passthrough API (Nathaniel Mott, July 25, 2021)</ref> | ||
<ref name="Skarredghost_Varjo"> | <ref name="Skarredghost_Varjo">[https://skarredghost.com/2022/06/08/varjo-xr-3-hands-on-review/ The Ghost Howls – Varjo XR‑3 hands‑on review] (June 8 2022)</ref> | ||
<ref name="Reddit_PicoMono">Reddit - Meta Quest Pro vs PICO 4 Passthrough Comparison (discussion summary, 2022)</ref> | <ref name="Reddit_PicoMono">Reddit - Meta Quest Pro vs PICO 4 Passthrough Comparison (discussion summary, 2022)</ref> | ||
<ref name="ViveXRElite">[https://www.vive.com/us/product/vive-xr-elite/overview/ HTC Vive XR Elite Product Page]</ref> | <ref name="ViveXRElite">[https://www.vive.com/us/product/vive-xr-elite/overview/ HTC Vive XR Elite Product Page]</ref> | ||
| Line 192: | Line 192: | ||
<ref name="VarjoXR4">[https://varjo.com/products/xr-4/ Varjo XR-4 Product Page]</ref> | <ref name="VarjoXR4">[https://varjo.com/products/xr-4/ Varjo XR-4 Product Page]</ref> | ||
<ref name="Auganix_Pico4Ultra">Auganix - Pico Unveils Pico 4 Ultra… (Aug 21, 2024)</ref> | <ref name="Auganix_Pico4Ultra">Auganix - Pico Unveils Pico 4 Ultra… (Aug 21, 2024)</ref> | ||
<ref name=" | <ref name="UploadVR_Q3Review">[https://www.uploadvr.com/quest-3-review/ UploadVR – Quest 3 Review: Excellent VR With Limited MR] (Oct 9 2023)</ref> | ||
<ref name="UploadVR_specs">UploadVR - Quest 3 Specs Compared To Quest 2 & Apple Vision Pro (David Heaney, Sep 27, 2023)</ref> | <ref name="UploadVR_specs">UploadVR - Quest 3 Specs Compared To Quest 2 & Apple Vision Pro (David Heaney, Sep 27, 2023)</ref> | ||
<ref name="UploadVR_ghosting">UploadVR - Quest 3 Review (on ghosting issue)</ref> | <ref name="UploadVR_ghosting">UploadVR - Quest 3 Review (on ghosting issue)</ref> | ||