Passthrough: Difference between revisions
Xinreality (talk | contribs) No edit summary |
Xinreality (talk | contribs) No edit summary |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 24: | Line 24: | ||
**The [[Meta Quest Pro]] (2022) was the first major consumer headset featuring high-quality, stereoscopic color passthrough, using a novel camera array (monochrome for depth/detail, RGB for color) and ML reconstruction.<ref name="MixedNews_Cambria"/> | **The [[Meta Quest Pro]] (2022) was the first major consumer headset featuring high-quality, stereoscopic color passthrough, using a novel camera array (monochrome for depth/detail, RGB for color) and ML reconstruction.<ref name="MixedNews_Cambria"/> | ||
**Competitors like the [[Pico 4]] (late 2022) and [[HTC Vive XR Elite]] (2023) also introduced color passthrough, although early implementations like the Pico 4's were initially monoscopic and lacked depth correction.<ref name="Reddit_PicoMono"></ref><ref name="ViveXRElite">[https://www.vive.com/us/product/vive-xr-elite/overview/ HTC Vive XR Elite Product Page]</ref> | **Competitors like the [[Pico 4]] (late 2022) and [[HTC Vive XR Elite]] (2023) also introduced color passthrough, although early implementations like the Pico 4's were initially monoscopic and lacked depth correction.<ref name="Reddit_PicoMono"></ref><ref name="ViveXRElite">[https://www.vive.com/us/product/vive-xr-elite/overview/ HTC Vive XR Elite Product Page]</ref> | ||
**Sony's [[PlayStation VR2]] (2023) included stereo passthrough, but kept it black-and-white, accessible via a dedicated button for quick checks.<ref name="RoadToVR_PSVR2"> | **Sony's [[PlayStation VR2]] (2023) included stereo passthrough, but kept it black-and-white, accessible via a dedicated button for quick checks.<ref name="RoadToVR_PSVR2"></ref> | ||
**The [[Meta Quest 3]] (late 2023) brought high-resolution stereo color passthrough with an active depth sensor (structured light projector) to the mainstream consumer market, offering significantly improved clarity and depth accuracy over Quest 2 and Quest Pro.<ref name="UploadVR_Q3Review"/><ref name="Quest3PassthroughReview">[https://www.roadtovr.com/meta-quest-3-review-vr-mixed-reality-ar/ RoadToVR Quest 3 Review detailing passthrough improvements]</ref> | **The [[Meta Quest 3]] (late 2023) brought high-resolution stereo color passthrough with an active depth sensor (structured light projector) to the mainstream consumer market, offering significantly improved clarity and depth accuracy over Quest 2 and Quest Pro.<ref name="UploadVR_Q3Review"/><ref name="Quest3PassthroughReview">[https://www.roadtovr.com/meta-quest-3-review-vr-mixed-reality-ar/ RoadToVR Quest 3 Review detailing passthrough improvements]</ref> | ||
**The [[Apple Vision Pro]] (2023 announcement, 2024 release) emphasized passthrough-based MR ("[[spatial computing]]"), using dual high-resolution color cameras, advanced processing ([[Apple R1]] chip), and a [[LiDAR]] scanner for precise depth mapping.<ref name="VisionProPassthrough">[https://www.apple.com/newsroom/2023/06/introducing-apple-vision-pro/ Apple Vision Pro Announcement]</ref><ref name="Verge_VisionPro">The Verge - Apple Vision Pro review: magic, until it’s not (Nilay Patel, June 2023)</ref> | **The [[Apple Vision Pro]] (2023 announcement, 2024 release) emphasized passthrough-based MR ("[[spatial computing]]"), using dual high-resolution color cameras, advanced processing ([[Apple R1]] chip), and a [[LiDAR]] scanner for precise depth mapping.<ref name="VisionProPassthrough">[https://www.apple.com/newsroom/2023/06/introducing-apple-vision-pro/ Apple Vision Pro Announcement]</ref><ref name="Verge_VisionPro">The Verge - Apple Vision Pro review: magic, until it’s not (Nilay Patel, June 2023)</ref> | ||
| Line 36: | Line 36: | ||
===Monochrome Passthrough=== | ===Monochrome Passthrough=== | ||
Uses black-and-white camera feeds. Common in earlier VR headsets (Oculus Rift S, Quest 1 & 2) or as a design choice (PSVR2), often leveraging existing grayscale tracking cameras.<ref name="UploadVR_Q3Review_MR"> | Uses black-and-white camera feeds. Common in earlier VR headsets (Oculus Rift S, Quest 1 & 2) or as a design choice (PSVR2), often leveraging existing grayscale tracking cameras.<ref name="UploadVR_Q3Review_MR"></ref><ref name="RoadToVR_PSVR2"/> Provides basic environmental awareness but lacks color cues and realism. Advantages include potentially better low-light sensitivity and lower processing requirements.<ref name="PCMag_passthrough"/> | ||
===Color Passthrough=== | ===Color Passthrough=== | ||
Uses [[RGB]] color cameras for a full-color view of the real world, greatly enhancing realism and enabling use cases like reading phone screens or interacting with colored objects. First widely available consumer example was Meta Quest Pro.<ref name="MixedNews_Cambria"/> Quality varies significantly based on camera resolution, processing, and calibration (for example Quest 3 offers ~10x the passthrough pixels of Quest 2).<ref name="UploadVR_specs"> | Uses [[RGB]] color cameras for a full-color view of the real world, greatly enhancing realism and enabling use cases like reading phone screens or interacting with colored objects. First widely available consumer example was Meta Quest Pro.<ref name="MixedNews_Cambria"/> Quality varies significantly based on camera resolution, processing, and calibration (for example Quest 3 offers ~10x the passthrough pixels of Quest 2).<ref name="UploadVR_specs"></ref> High-quality color passthrough (for example Varjo XR series, Vision Pro) aims for near-photorealism.<ref name="Skarredghost_Varjo"/><ref name="VisionProPassthrough"/> Requires more powerful hardware and sophisticated software. | ||
===Monoscopic vs. Stereoscopic=== | ===Monoscopic vs. Stereoscopic=== | ||
| Line 66: | Line 66: | ||
Creating high-quality, comfortable passthrough involves overcoming significant hurdles: | Creating high-quality, comfortable passthrough involves overcoming significant hurdles: | ||
*'''[[Latency (engineering)|Latency]]:''' The delay between real-world motion and the passthrough display update ([[photon-to-photon latency]]). High latency (>~20ms<ref name="LatencyThreshold"/>) causes disorientation, [[motion sickness]] ("world swimming"), and breaks immersion. Fast processing pipelines are essential.<ref name="PassthroughChallengesUploadVR"/> Residual latency can cause ghosting or trailing artifacts on moving objects.<ref name="UploadVR_ghosting"> | *'''[[Latency (engineering)|Latency]]:''' The delay between real-world motion and the passthrough display update ([[photon-to-photon latency]]). High latency (>~20ms<ref name="LatencyThreshold"/>) causes disorientation, [[motion sickness]] ("world swimming"), and breaks immersion. Fast processing pipelines are essential.<ref name="PassthroughChallengesUploadVR"/> Residual latency can cause ghosting or trailing artifacts on moving objects.<ref name="UploadVR_ghosting"></ref> | ||
*'''Resolution and Image Quality:''' Camera feeds are often lower resolution than human vision, leading to pixelation or blurriness, making fine details (like text) hard to see.<ref name="CameraLimitations">[https://arstechnica.com/gadgets/2023/10/quest-3-review-finally-real-mixed-reality-for-under-500/ Ars Technica Quest 3 Review discussing passthrough quality]</ref> Limited [[dynamic range]] struggles with bright highlights and dark shadows compared to the human eye. Poor [[low-light performance]] results in noisy, grainy images.<ref name="CameraLimitations"/> Achieving high resolution and good image quality requires better sensors and significant processing power. | *'''Resolution and Image Quality:''' Camera feeds are often lower resolution than human vision, leading to pixelation or blurriness, making fine details (like text) hard to see.<ref name="CameraLimitations">[https://arstechnica.com/gadgets/2023/10/quest-3-review-finally-real-mixed-reality-for-under-500/ Ars Technica Quest 3 Review discussing passthrough quality]</ref> Limited [[dynamic range]] struggles with bright highlights and dark shadows compared to the human eye. Poor [[low-light performance]] results in noisy, grainy images.<ref name="CameraLimitations"/> Achieving high resolution and good image quality requires better sensors and significant processing power. | ||
*'''Camera Placement and Perspective Mismatch:''' Cameras are offset from the user's eyes, causing [[parallax]] errors if not corrected. Naive display leads to distorted views, incorrect scale, and depth perception issues, especially for close objects.<ref name="PassthroughChallengesUploadVR"/> Sophisticated [[reprojection]] algorithms are needed to warp the camera view to match the eye's perspective, but perfect correction is difficult.<ref name="KGuttag_Align"/> This geometric misalignment can cause eye strain or discomfort.<ref name="KGuttag_Align"/> Close objects (<~0.5m) often appear warped even in good systems due to sensor/lens limitations and reprojection challenges.<ref name="UploadVR_Q3Review_MR"/> | *'''Camera Placement and Perspective Mismatch:''' Cameras are offset from the user's eyes, causing [[parallax]] errors if not corrected. Naive display leads to distorted views, incorrect scale, and depth perception issues, especially for close objects.<ref name="PassthroughChallengesUploadVR"/> Sophisticated [[reprojection]] algorithms are needed to warp the camera view to match the eye's perspective, but perfect correction is difficult.<ref name="KGuttag_Align"/> This geometric misalignment can cause eye strain or discomfort.<ref name="KGuttag_Align"/> Close objects (<~0.5m) often appear warped even in good systems due to sensor/lens limitations and reprojection challenges.<ref name="UploadVR_Q3Review_MR"/> | ||
| Line 175: | Line 175: | ||
<ref name="KGuttag_Align">[https://kguttag.com/2023/09/26/apple-vision-pro-part-6-passthrough-mixed-reality-ptmr-problems/ KGOnTech – Perspective‑Correct Passthrough] (Sept 26 2023)</ref> | <ref name="KGuttag_Align">[https://kguttag.com/2023/09/26/apple-vision-pro-part-6-passthrough-mixed-reality-ptmr-problems/ KGOnTech – Perspective‑Correct Passthrough] (Sept 26 2023)</ref> | ||
<ref name="QuestProPassthrough">[https://www.meta.com/blog/quest/meta-reality-passthrough-quest-pro/ Meta Blog: Inside Meta Reality and Passthrough on Quest Pro]</ref> | <ref name="QuestProPassthrough">[https://www.meta.com/blog/quest/meta-reality-passthrough-quest-pro/ Meta Blog: Inside Meta Reality and Passthrough on Quest Pro]</ref> | ||
<ref name="UploadVR_Q3Review">UploadVR - | <ref name="UploadVR_Q3Review">[https://www.uploadvr.com/quest-3-review/ UploadVR – Quest 3 Review: Excellent VR With Limited MR] (Oct 9 2023)</ref> | ||
<ref name="UploadVR_Q3Review_MR">[https://www.uploadvr.com/quest-3-review/ UploadVR – Quest 3 Review: Excellent VR With Limited MR] (Oct 9 2023)</ref> | |||
<ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref> | <ref name="LatencyThreshold">[https://research.nvidia.com/publication/2016-07_Latency-Requirements-Plausible-Interaction-Augmented-and-Virtual-Reality Latency Requirements for Plausible Interaction in Augmented and Virtual Reality] - Research discussing latency impact.</ref> | ||
<ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> | <ref name="MilgramKishino1994">Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1321-1329.</ref> | ||
| Line 185: | Line 186: | ||
<ref name="Reddit_PicoMono">[https://www.reddit.com/r/virtualreality/comments/ynz7yv/meta_quest_pro_vs_pico_4_passthrough_comparison/ Reddit – Meta Quest Pro vs PICO 4 Passthrough Comparison] (2022 thread)</ref> | <ref name="Reddit_PicoMono">[https://www.reddit.com/r/virtualreality/comments/ynz7yv/meta_quest_pro_vs_pico_4_passthrough_comparison/ Reddit – Meta Quest Pro vs PICO 4 Passthrough Comparison] (2022 thread)</ref> | ||
<ref name="ViveXRElite">[https://www.vive.com/us/product/vive-xr-elite/overview/ HTC Vive XR Elite Product Page]</ref> | <ref name="ViveXRElite">[https://www.vive.com/us/product/vive-xr-elite/overview/ HTC Vive XR Elite Product Page]</ref> | ||
<ref name="RoadToVR_PSVR2"> | <ref name="RoadToVR_PSVR2">[https://www.roadtovr.com/psvr-2-review/ Road to VR – PSVR 2 Review] (Feb 22 2023)</ref> | ||
<ref name="Quest3PassthroughReview">[https://www.roadtovr.com/meta-quest-3-review-vr-mixed-reality-ar/ RoadToVR Quest 3 Review detailing passthrough improvements]</ref> | <ref name="Quest3PassthroughReview">[https://www.roadtovr.com/meta-quest-3-review-vr-mixed-reality-ar/ RoadToVR Quest 3 Review detailing passthrough improvements]</ref> | ||
<ref name="VisionProPassthrough">[https://www.apple.com/newsroom/2023/06/introducing-apple-vision-pro/ Apple Vision Pro Announcement]</ref> | <ref name="VisionProPassthrough">[https://www.apple.com/newsroom/2023/06/introducing-apple-vision-pro/ Apple Vision Pro Announcement]</ref> | ||
| Line 193: | Line 194: | ||
<ref name="Auganix_Pico4Ultra">[https://www.auganix.org/vr-news-pico-launches-pico-4-ultra/ Auganix – Pico Unveils ‘Pico 4 Ultra’] (Aug 21 2024)</ref> | <ref name="Auganix_Pico4Ultra">[https://www.auganix.org/vr-news-pico-launches-pico-4-ultra/ Auganix – Pico Unveils ‘Pico 4 Ultra’] (Aug 21 2024)</ref> | ||
<ref name="UploadVR_Q3Review">[https://www.uploadvr.com/quest-3-review/ UploadVR – Quest 3 Review: Excellent VR With Limited MR] (Oct 9 2023)</ref> | <ref name="UploadVR_Q3Review">[https://www.uploadvr.com/quest-3-review/ UploadVR – Quest 3 Review: Excellent VR With Limited MR] (Oct 9 2023)</ref> | ||
<ref name="UploadVR_specs"> | <ref name="UploadVR_specs">[https://www.uploadvr.com/quest-3-specs/ UploadVR – Quest 3 Specs Compared to Quest 2 & Apple Vision Pro] (Sept 27 2023)</ref> | ||
<ref name="UploadVR_ghosting"> | <ref name="UploadVR_ghosting">[https://www.uploadvr.com/quest-3-review/#passthrough Ghosting issue noted in UploadVR Quest 3 Review] (Oct 9 2023)</ref> | ||
<ref name="CameraLimitations">[https://arstechnica.com/gadgets/2023/10/quest-3-review-finally-real-mixed-reality-for-under-500/ Ars Technica Quest 3 Review discussing passthrough quality]</ref> | <ref name="CameraLimitations">[https://arstechnica.com/gadgets/2023/10/quest-3-review-finally-real-mixed-reality-for-under-500/ Ars Technica Quest 3 Review discussing passthrough quality]</ref> | ||
<ref name="Varjo_blog">Varjo Blog - Video Pass-Through XR - Merge Real and Virtual (Urho Konttori, 2020)</ref> | <ref name="Varjo_blog">Varjo Blog - Video Pass-Through XR - Merge Real and Virtual (Urho Konttori, 2020)</ref> | ||