Occlusion

Occlusion happens when one object in a three-dimensional scene blocks another object from view. The word covers several related ideas in virtual reality and augmented reality that are easy to confuse because they all turn on the same basic fact: a nearer thing hides a farther thing. In human vision, occlusion is one of the cues the brain uses to judge depth. In rendering, it is the property a system must respect so that virtual content sits correctly among real and other virtual objects. In positional tracking, occlusion is a failure mode that happens when a tracked object is hidden from a sensor. A separate but analogous effect appears in audio, where solid geometry between a sound source and a listener muffles the sound.

Occlusion as a depth cue

Occlusion, also called interposition, is a monocular depth cue, meaning it works with a single eye and does not depend on binocular disparity. When one object obscures part of another, the brain interprets the obscured object as being farther away. If a tree blocks part of a mountain behind it, the viewer reads the tree as the nearer object even with one eye closed, because mountains are not usually torn open by trees.

This cue is powerful because it gives clear ordering information: it tells the observer which object is in front, even though on its own it does not say how far apart the two objects are. It works alongside other monocular cues such as relative size, texture gradient, linear perspective, and motion parallax. For VR and AR, the practical consequence is that the renderer has to honor occlusion ordering exactly. If a closer object is drawn so that a farther object shows through it, the scene stops reading as a coherent space, because the strongest everyday cue for "what is in front" has been violated.

Rendering occlusion in AR and mixed reality

The hardest version of the problem appears in augmented reality and mixed reality, where virtual objects are composited into a view of the real world. For the result to look believable, virtual objects must be able to pass behind real ones: a virtual character should disappear when it walks behind a real sofa and reappear as the user moves. Google describes occlusion plainly as "the ability for digital objects to accurately appear in front of or behind real world objects."

Why it is hard

A pure overlay system has no idea where real surfaces are, so it draws every virtual pixel on top of the camera feed. The virtual object then floats in front of everything, including real objects that are physically nearer, and the illusion that it occupies the room collapses. Solving occlusion therefore requires the system to sense the geometry of the real world and compare its distance to the virtual content for every pixel on screen.

How systems sense real-world depth

Several methods give a headset or phone the depth information it needs:

Approach	How it works
Depth sensing hardware	Dedicated sensors such as time-of-flight cameras, LiDAR scanners, or stereo cameras measure the distance to real surfaces directly and build a depth map of the scene.
Passthrough and depth from motion	Cameras that already feed the user a view of the room can also be used to estimate depth. Google's ARCore Depth API runs a depth-from-motion algorithm that captures multiple images from slightly different viewpoints as the device moves and compares them to estimate the distance to every pixel, with no dedicated depth sensor required.
Scene meshing	The system reconstructs a polygon mesh of the surrounding surfaces. Apple's ARKit Scene Geometry API generates a 3D mesh of the environment that apps can test against, and ARKit also offers People Occlusion, which lets virtual content pass behind people detected in the camera image.
Semantic segmentation	Machine learning classifies camera pixels to recognize objects such as people, walls, or furniture, which can supply or refine occlusion even on devices without depth sensors.

Once depth is known, occlusion is handled in the renderer by comparing depth values. Where the virtual object is closer than the real surface at a given pixel, the virtual color is drawn; where the real surface is closer, the camera pixel is kept and the virtual pixel is discarded. This is the same depth-buffer logic used in ordinary 3D graphics, extended so that the real world acts as one more set of depth values.

Dynamic and static occlusion

A useful distinction is between static and dynamic occlusion. Static occlusion deals with parts of the scene that do not move, such as walls, floors, and furniture, whose geometry can be captured once and reused. Dynamic occlusion deals with things that move during use, most importantly people and the user's own hands. Moving occluders are harder because their depth has to be re-estimated continuously and any lag shows up as virtual content briefly bleeding through a real object, or a real object failing to hide the virtual one. The release of the ARCore Depth API in 2020 was significant partly because it brought real-time, dynamic occlusion to a large base of phones that have no depth sensor at all.

Tracking occlusion

A different sense of the word matters in positional tracking. In Outside-in tracking, one or more cameras or sensors are placed in fixed positions in the room and look toward the headset and controllers. Because these systems work by line of sight, they are vulnerable to occlusion: when a tracked object is blocked from a sensor's view by the user's body, by furniture, or by another tracked object, the sensor can no longer see it and tracking of that object degrades or is lost.

This is why outside-in setups are usually arranged to cover the play space from more than one angle, so that an object hidden from one sensor is still visible to another. It is also one of the reasons inside-out tracking, where the cameras sit on the headset and look outward, became popular: moving the cameras onto the user removes the specific failure where the user's own body steps between an external camera and a controller. Tracking occlusion is purely a sensing problem and is unrelated to how the final image is drawn; the two senses of "occlusion" share only the underlying idea that something nearer is blocking the view of something farther. For more on the general topic, see Tracking.

Audio occlusion

Occlusion has a direct counterpart in sound. In 3D audio, audio occlusion is the effect of solid geometry sitting between a sound source and the listener, which lowers the volume and filters the sound. Because high frequencies are attenuated more than low frequencies, an occluded source sounds muffled, the same way a voice behind a closed door does. Spatial audio engines typically test the path from the listener to each source against scene geometry, then apply a low-pass filter and a volume reduction when the path is blocked, with the amount depending on the material and thickness of the obstruction. Handling audio occlusion correctly reinforces the same sense of a shared space that visual occlusion creates, because sounds stop passing through walls at full volume.

See also

• Augmented reality
• Passthrough
• Depth sensing
• Outside-in tracking
• Positional tracking
• 3D audio