Spatial computing: Difference between revisions
Xinreality (talk | contribs) No edit summary |
Xinreality (talk | contribs) m Text replacement - "e.g.," to "for example" Tags: Mobile edit Mobile web edit |
||
| Line 23: | Line 23: | ||
Spatial computing typically involves several key components working together: | Spatial computing typically involves several key components working together: | ||
* '''Machine Perception of Space:''' Devices must understand the physical environment in 3D. This involves technologies like [[Simultaneous Localization and Mapping]] (SLAM) to track the device's position and orientation while building a map of the space.<ref name="DurrantWhyteSLAM"/> [[Depth sensor]]s (like [[LiDAR]] or Time-of-Flight cameras) and [[RGB camera]]s capture geometric and visual information. [[Computer vision]] algorithms, often powered by [[artificial intelligence]] (AI), interpret this data to recognize surfaces, objects ( | * '''Machine Perception of Space:''' Devices must understand the physical environment in 3D. This involves technologies like [[Simultaneous Localization and Mapping]] (SLAM) to track the device's position and orientation while building a map of the space.<ref name="DurrantWhyteSLAM"/> [[Depth sensor]]s (like [[LiDAR]] or Time-of-Flight cameras) and [[RGB camera]]s capture geometric and visual information. [[Computer vision]] algorithms, often powered by [[artificial intelligence]] (AI), interpret this data to recognize surfaces, objects (for example walls, tables, chairs), people, and potentially understand scene semantics.<ref name="CogentSLAM"/><ref name="TechTargetWhatIs"/> | ||
* '''Persistence and Context:''' Digital objects or information placed within the spatial environment can maintain their position and state relative to the physical world, even when the user looks away or leaves and returns (spatial anchors). The system uses its understanding of spatial context to anchor digital elements appropriately and realistically, potentially enabling occlusion (virtual objects appearing behind real ones) and physics interactions.<ref name="HandwikiHistory"/> | * '''Persistence and Context:''' Digital objects or information placed within the spatial environment can maintain their position and state relative to the physical world, even when the user looks away or leaves and returns (spatial anchors). The system uses its understanding of spatial context to anchor digital elements appropriately and realistically, potentially enabling occlusion (virtual objects appearing behind real ones) and physics interactions.<ref name="HandwikiHistory"/> | ||
* '''Natural User Interaction:''' Input moves beyond the [[keyboard]] and [[mouse]]. Common interaction methods include [[Hand tracking]] (recognizing hand shapes and gestures), [[Eye tracking]] (using gaze as a pointer or input trigger), [[Voice command]]s, and sometimes specialized controllers. The goal is intuitive interaction that mimics how humans interact with the physical world, making the computer interface feel "invisible."<ref name="PCMagWhatIs"/><ref name="Microsoft HoloLens"/> | * '''Natural User Interaction:''' Input moves beyond the [[keyboard]] and [[mouse]]. Common interaction methods include [[Hand tracking]] (recognizing hand shapes and gestures), [[Eye tracking]] (using gaze as a pointer or input trigger), [[Voice command]]s, and sometimes specialized controllers. The goal is intuitive interaction that mimics how humans interact with the physical world, making the computer interface feel "invisible."<ref name="PCMagWhatIs"/><ref name="Microsoft HoloLens"/> | ||
| Line 35: | Line 35: | ||
* '''Displays:''' High-resolution, high-refresh-rate micro-displays ([[Micro-OLED]], [[MicroLED]]) for rendering sharp images. [[Waveguide (optics)|Waveguides]] or other novel optics are used in optical see-through AR glasses. Wide [[Field of view (computer vision)|field-of-view]] (FOV) lenses are common in VR/MR headsets. | * '''Displays:''' High-resolution, high-refresh-rate micro-displays ([[Micro-OLED]], [[MicroLED]]) for rendering sharp images. [[Waveguide (optics)|Waveguides]] or other novel optics are used in optical see-through AR glasses. Wide [[Field of view (computer vision)|field-of-view]] (FOV) lenses are common in VR/MR headsets. | ||
* '''Processing Units:''' Powerful, energy-efficient [[System on a chip|Systems-on-Chip]] (SoCs) with strong CPUs, GPUs, and often dedicated AI/[[Neural processing unit|NPU]]s or co-processors (like Apple's R1 chip<ref name="VisionProAnnounce"/>) handle complex sensor fusion, computer vision tasks, and real-time [[3D rendering]] on-device. | * '''Processing Units:''' Powerful, energy-efficient [[System on a chip|Systems-on-Chip]] (SoCs) with strong CPUs, GPUs, and often dedicated AI/[[Neural processing unit|NPU]]s or co-processors (like Apple's R1 chip<ref name="VisionProAnnounce"/>) handle complex sensor fusion, computer vision tasks, and real-time [[3D rendering]] on-device. | ||
* '''Input Devices:''' Beyond integrated tracking (hand, eye, voice), some systems use handheld [[Controller (computing)|controllers]] ( | * '''Input Devices:''' Beyond integrated tracking (hand, eye, voice), some systems use handheld [[Controller (computing)|controllers]] (for example Meta Quest controllers) providing buttons, joysticks, and [[haptic feedback]]. | ||
=== Software === | === Software === | ||
* '''[[Spatial mapping]] Algorithms:''' Primarily SLAM and related techniques ( | * '''[[Spatial mapping]] Algorithms:''' Primarily SLAM and related techniques (for example visual-inertial odometry) to create real-time 3D environmental maps and track device pose.<ref name="DurrantWhyteSLAM"/> | ||
* '''[[Computer vision]] & [[Artificial intelligence|AI]]/[[Machine learning|ML]]:''' Algorithms for object recognition, [[Gesture recognition|gesture detection]], scene understanding, [[semantic segmentation]], user intent prediction, and optimizing rendering.<ref name="TechTargetWhatIs"/> | * '''[[Computer vision]] & [[Artificial intelligence|AI]]/[[Machine learning|ML]]:''' Algorithms for object recognition, [[Gesture recognition|gesture detection]], scene understanding, [[semantic segmentation]], user intent prediction, and optimizing rendering.<ref name="TechTargetWhatIs"/> | ||
* '''[[Rendering engine|Rendering Engines]]:''' Tools like [[Unity (game engine)|Unity]] and [[Unreal Engine]] provide frameworks for developing 3D environments, handling physics, and supporting AR/VR application development.<ref name="UnityRef"/> | * '''[[Rendering engine|Rendering Engines]]:''' Tools like [[Unity (game engine)|Unity]] and [[Unreal Engine]] provide frameworks for developing 3D environments, handling physics, and supporting AR/VR application development.<ref name="UnityRef"/> | ||
* '''[[Operating system|Operating Systems]] & [[Software development kit|SDKs]]:''' Specialized OSs ( | * '''[[Operating system|Operating Systems]] & [[Software development kit|SDKs]]:''' Specialized OSs (for example Apple [[visionOS]], [[Windows Holographic]], [[Android]] variants) manage spatial tasks. SDKs (for example [[ARKit]], [[ARCore]], [[OpenXR]], MRTK) provide APIs for developers to build spatial applications. | ||
* '''[[Cloud computing|Cloud]] and [[Edge computing]]:''' Used to offload heavy computation (rendering, AI processing, large-scale mapping), enable collaborative multi-user experiences ( | * '''[[Cloud computing|Cloud]] and [[Edge computing]]:''' Used to offload heavy computation (rendering, AI processing, large-scale mapping), enable collaborative multi-user experiences (for example shared spatial anchors, "AR Cloud" concepts), and stream content.<ref name="NvidiaSpatialCloud"/> | ||
* '''Connectivity:''' High-bandwidth, low-latency wireless like [[Wi-Fi 6E]] and [[5G]] are crucial for tetherless experiences and cloud/edge reliance. | * '''Connectivity:''' High-bandwidth, low-latency wireless like [[Wi-Fi 6E]] and [[5G]] are crucial for tetherless experiences and cloud/edge reliance. | ||
| Line 48: | Line 48: | ||
Spatial computing is a foundational concept enabling advanced forms of VR, AR, and MR (often grouped under the umbrella term [[Extended Reality|XR]]). While closely related and sometimes used interchangeably in marketing, there are nuances: | Spatial computing is a foundational concept enabling advanced forms of VR, AR, and MR (often grouped under the umbrella term [[Extended Reality|XR]]). While closely related and sometimes used interchangeably in marketing, there are nuances: | ||
* '''[[Virtual Reality]] (VR):''' Creates a fully immersive digital environment replacing the user's real-world view. Spatial computing principles apply ''within'' this virtual space for tracking user movement (room-scale VR), environmental awareness ( | * '''[[Virtual Reality]] (VR):''' Creates a fully immersive digital environment replacing the user's real-world view. Spatial computing principles apply ''within'' this virtual space for tracking user movement (room-scale VR), environmental awareness (for example safety boundaries based on real walls), and interacting with virtual objects using tracked hands or controllers. | ||
* '''[[Augmented Reality]] (AR):''' Overlays digital information onto the real world, typically via smartphones, tablets, or simpler smart glasses. Interaction might be basic. Mobile AR uses spatial computing for plane detection and tracking but often lacks deep environmental understanding. | * '''[[Augmented Reality]] (AR):''' Overlays digital information onto the real world, typically via smartphones, tablets, or simpler smart glasses. Interaction might be basic. Mobile AR uses spatial computing for plane detection and tracking but often lacks deep environmental understanding. | ||
* '''[[Mixed Reality]] (MR):''' A more advanced form of AR where digital objects are integrated more realistically into the physical environment, appearing anchored to and potentially interacting with real surfaces and objects. Users can interact with both physical and virtual elements simultaneously. MR heavily relies on sophisticated spatial computing for real-time mapping, understanding, occlusion, and interaction. Headsets like HoloLens, Magic Leap, and passthrough devices like Vision Pro and Quest 3 are often categorized as MR. | * '''[[Mixed Reality]] (MR):''' A more advanced form of AR where digital objects are integrated more realistically into the physical environment, appearing anchored to and potentially interacting with real surfaces and objects. Users can interact with both physical and virtual elements simultaneously. MR heavily relies on sophisticated spatial computing for real-time mapping, understanding, occlusion, and interaction. Headsets like HoloLens, Magic Leap, and passthrough devices like Vision Pro and Quest 3 are often categorized as MR. | ||
| Line 57: | Line 57: | ||
Spatial computing builds upon and overlaps with several earlier computing paradigms: | Spatial computing builds upon and overlaps with several earlier computing paradigms: | ||
* '''[[Ubiquitous computing]] (Pervasive Computing):''' Envisions computers embedded everywhere, becoming invisible parts of daily life (Mark Weiser's vision). Spatial computing shares the goal of moving computation beyond the desktop, but specifically focuses on 3D spatial awareness and interaction, whereas ubiquitous computing is broader ( | * '''[[Ubiquitous computing]] (Pervasive Computing):''' Envisions computers embedded everywhere, becoming invisible parts of daily life (Mark Weiser's vision). Spatial computing shares the goal of moving computation beyond the desktop, but specifically focuses on 3D spatial awareness and interaction, whereas ubiquitous computing is broader (for example smart home devices). Wearable spatial devices like AR glasses align with the ubiquitous vision.<ref name="HandwikiHistory"/> | ||
* '''[[Ambient computing]]:''' Often used interchangeably with ubiquitous computing, emphasizing calm, background operation responsive to user presence, often without traditional screens ( | * '''[[Ambient computing]]:''' Often used interchangeably with ubiquitous computing, emphasizing calm, background operation responsive to user presence, often without traditional screens (for example smart speakers, automated lighting). Spatial computing can be ambient (for example AR glasses providing subtle cues), but often involves explicit visual overlays, contrasting with ambient computing's typical emphasis on screenlessness.<ref name="ArgoDesign Medium"/> | ||
* '''[[Context-aware computing]]:''' Systems that adapt based on current context (location, time, user activity). Spatial computing is inherently context-aware, focusing specifically on real-time ''spatial'' context (geometry, pose, environment). While any context-aware app uses context ( | * '''[[Context-aware computing]]:''' Systems that adapt based on current context (location, time, user activity). Spatial computing is inherently context-aware, focusing specifically on real-time ''spatial'' context (geometry, pose, environment). While any context-aware app uses context (for example GPS location), spatial computing requires understanding and interaction within the 3D physical environment.<ref name="HandwikiHistory"/> | ||
In summary, spatial computing systems are typically context-aware and can be part of ubiquitous/ambient computing scenarios. Its differentiator is the requirement for real-time 3D spatial understanding and interaction, blending digital content directly into the user's perceived physical space. | In summary, spatial computing systems are typically context-aware and can be part of ubiquitous/ambient computing scenarios. Its differentiator is the requirement for real-time 3D spatial understanding and interaction, blending digital content directly into the user's perceived physical space. | ||
| Line 69: | Line 69: | ||
* '''Healthcare:''' [[Surgical planning]] using 3D patient models, AR overlays during surgery for navigation<ref name="ChenAR Surgery"/>, immersive medical training simulations, [[Physical therapy|rehabilitation]] exercises using AR/VR, visualizing complex medical data (MRI/CT scans) in 3D.<ref name="SpatialHealthcare"/> | * '''Healthcare:''' [[Surgical planning]] using 3D patient models, AR overlays during surgery for navigation<ref name="ChenAR Surgery"/>, immersive medical training simulations, [[Physical therapy|rehabilitation]] exercises using AR/VR, visualizing complex medical data (MRI/CT scans) in 3D.<ref name="SpatialHealthcare"/> | ||
* '''Education and Training:''' Immersive learning experiences (virtual field trips, science labs), visualizing complex concepts (molecules, historical events) in 3D, complex task training (aircraft maintenance, emergency response) with AR guidance.<ref name="BaccaAR Education"/> | * '''Education and Training:''' Immersive learning experiences (virtual field trips, science labs), visualizing complex concepts (molecules, historical events) in 3D, complex task training (aircraft maintenance, emergency response) with AR guidance.<ref name="BaccaAR Education"/> | ||
* '''Collaboration and Communication:''' Virtual meetings with spatial presence ([[avatar]]s in shared spaces), remote collaboration on 3D projects, shared digital workspaces ( | * '''Collaboration and Communication:''' Virtual meetings with spatial presence ([[avatar]]s in shared spaces), remote collaboration on 3D projects, shared digital workspaces (for example virtual whiteboards, multiple virtual monitors).<ref name="Spatial Collaboration"/> | ||
* '''Retail and E-commerce:''' Virtually trying on clothes or accessories (AR mirrors), placing virtual furniture or appliances in a room using mobile AR apps before purchase.<ref name="IKEA"/> | * '''Retail and E-commerce:''' Virtually trying on clothes or accessories (AR mirrors), placing virtual furniture or appliances in a room using mobile AR apps before purchase.<ref name="IKEA"/> | ||
* '''Entertainment and Gaming:''' Highly immersive VR games with room-scale tracking, location-based AR games blending virtual elements with the real world, interactive spatial storytelling, spatial viewing of 360°/[[Volumetric video|volumetric]] content.<ref name="PokemonGoRef"/> | * '''Entertainment and Gaming:''' Highly immersive VR games with room-scale tracking, location-based AR games blending virtual elements with the real world, interactive spatial storytelling, spatial viewing of 360°/[[Volumetric video|volumetric]] content.<ref name="PokemonGoRef"/> | ||
* '''Navigation and Information Access:''' Contextual information overlaid on the real world ( | * '''Navigation and Information Access:''' Contextual information overlaid on the real world (for example AR directions in streets or airports, information about landmarks), indoor navigation aids. | ||
* '''Architecture and Construction:''' Visualizing architectural designs on-site using AR, virtual walkthroughs of buildings in VR before construction.<ref name="WangAR Construction"/> | * '''Architecture and Construction:''' Visualizing architectural designs on-site using AR, virtual walkthroughs of buildings in VR before construction.<ref name="WangAR Construction"/> | ||
| Line 102: | Line 102: | ||
* '''Convergence:''' Further blending with [[Internet of Things|IoT]], [[Cloud computing]], [[Edge computing]], and potentially forming key infrastructure for concepts like the [[Metaverse]]. | * '''Convergence:''' Further blending with [[Internet of Things|IoT]], [[Cloud computing]], [[Edge computing]], and potentially forming key infrastructure for concepts like the [[Metaverse]]. | ||
* '''Accessibility:''' Lower price points over time driving wider consumer and enterprise adoption. | * '''Accessibility:''' Lower price points over time driving wider consumer and enterprise adoption. | ||
* '''Enhanced Interaction:''' Advances in [[Brain–computer interface|brain-computer interfaces]] or sophisticated sensor-based inputs ( | * '''Enhanced Interaction:''' Advances in [[Brain–computer interface|brain-computer interfaces]] or sophisticated sensor-based inputs (for example EMG wristbands<ref name="MetaEMG"/>) could offer new ways to interact spatially. | ||
Technology leaders like Tim Cook see it as profoundly changing human-computer interaction.<ref name="9to5MacCookMemo"/> Futurists like Cathy Hackl frame it as the next computing wave enabling new forms of communication and machine intelligence.<ref name="HacklIndependent"/> Microsoft emphasizes productivity gains,<ref name="KipmanMR"/> while Meta focuses on social connection in the metaverse. The long-term vision often involves seamlessly blending digital information and interaction into our everyday perception of the physical world. | Technology leaders like Tim Cook see it as profoundly changing human-computer interaction.<ref name="9to5MacCookMemo"/> Futurists like Cathy Hackl frame it as the next computing wave enabling new forms of communication and machine intelligence.<ref name="HacklIndependent"/> Microsoft emphasizes productivity gains,<ref name="KipmanMR"/> while Meta focuses on social connection in the metaverse. The long-term vision often involves seamlessly blending digital information and interaction into our everyday perception of the physical world. | ||