3D audio

See also: VR audio

3D audio, also called spatial audio, is the reproduction of sound so that a listener perceives each source as coming from a specific point or direction in three-dimensional space, including above, below, and behind the head. In Virtual Reality and Augmented Reality, 3D audio is rendered in real time and locked to the virtual world: as the user turns their head, the rendering updates so that a sound keeps coming from its fixed position in the scene rather than from a fixed position relative to the ears. This head-tracked, world-anchored behavior is what separates VR spatial audio from ordinary stereo or surround sound, and it is a major contributor to Immersion and Presence.

Why 3D audio matters for VR and AR

On a flat monitor a player has a limited field of view and few reliable cues about distance, so sound mostly tells direction rather than position. In VR, audio can carry the same spatial information the eyes do: a footstep behind the listener, a voice to the left, a machine humming somewhere overhead. Because the headset already tracks head pose for the display, that same tracking data can drive the audio, keeping sources stable in world space while the user looks around.

Early VR developers treated this as central rather than decorative. At CES 2015 Oculus VR demonstrated a licensed 3D audio engine, RealSpace 3D from VisiSonics, using a pair of adjustable on-ear headphones attached to the HMD; people who tried it reported a clear jump in immersion. Oculus chief scientist Michael Abrash described 3D sound not as an addition to VR but as a multiplier, because spatialized cues help a person orient inside the environment in a way a screen cannot. Modern headsets fold this capability into their core software development kits, and head-locked spatial audio is now standard across the major VR and AR platforms.

Binaural audio

Load video

YouTube

YouTube might collect personal data. Privacy Policy

ContinueDismiss

Binaural recording in New York City

Humans localize sound using two ears. The brain compares the tiny time difference between when a sound reaches the near ear versus the far ear (the interaural time difference, or ITD) and the loudness difference caused by the head blocking sound (the interaural level difference, or ILD), then adds the spectral coloring produced by the outer ear, head, and torso to resolve direction, including front versus back and up versus down.

The classic way to capture this is binaural recording, in which two microphones are placed about as far apart as a pair of human ears, often inside a dummy head with molded ear shapes so the recording picks up the same head shadow and ear filtering a listener would experience. The technique is old: in 1881 Clement Ader demonstrated the Theatrophone at the International Exposition of Electricity in Paris, using rows of telephone transmitters arranged across the front of a stage, with separate left and right channels sent to each ear, to give listeners a stereophonic impression of opera performances transmitted from the Paris Opera. It is regarded as one of the first two-channel audio systems.

Binaural playback works correctly only over headphones or a tightly arranged stereo pair. Ordinary loudspeakers suffer from crosstalk, where sound meant for the left ear also reaches the right ear and vice versa, which cancels much of the binaural effect; speakers have to sit close together, roughly within 10 to 30 degrees, to keep crosstalk low. Headphones avoid the problem because each ear hears only its own channel. The effect survives the fact that no single pair of headphones fits everyone, since ear and head anatomy varies a great deal between people, so even average earbuds reproduce a convincing binaural image.

The head-related transfer function

The head-related transfer function (HRTF) is the mathematical description of how an incoming sound is filtered by a person's head, torso, and outer ears before it reaches each eardrum. It is a pair of filters, one per ear, that changes with the direction of the source, and it encodes the direction-dependent cues the brain uses for localization, externalization (hearing a sound as outside the head rather than inside it), and a sense of realism.

A VR engine spatializes a mono sound by convolving it with the HRTF that matches the source's current direction relative to the head, producing a left and right signal that the brain interprets as coming from that point in space. Because the HRTF set is indexed by direction, the same machinery makes head tracking straightforward: the system reads head orientation from the tracking sensors and, frame by frame, picks new HRTFs and adjusts ITD and ILD so a source that was in front of the user moves to the left ear when the user turns to the right, exactly as it would in the real world.

HRTF personalization

Most platforms ship a generic HRTF measured on a dummy head or averaged across many people, which works well enough for a strong spatial impression. Using an HRTF individualized to the specific listener improves perceived localization, front-back discrimination, externalization, tonal accuracy, and overall realism, particularly for elevation (sources in the vertical, or sagittal, plane). The catch is that measuring an individual HRTF traditionally requires a specialized acoustic setup, so vendors have pursued shortcuts such as selecting from a small library of measured profiles or estimating a profile from photos or video of the listener's ears.

Ambisonics

Load video

YouTube

YouTube might collect personal data. Privacy Policy

ContinueDismiss

RealSpace 3D Audio demo

Ambisonics is a full-sphere surround format that represents a whole sound field rather than a fixed set of speaker feeds, which makes it well suited to 360-degree video and VR because the field can be rotated to follow head movement and then decoded to headphones through HRTFs. First-order ambisonics uses four channels, known together as B-format: W carries the omnidirectional component, while X, Y, and Z capture the front-back, left-right, and up-down components like three figure-of-eight microphones.

Ambisonics is hierarchical: each higher order adds channels and increases spatial resolution. Second order uses 9 channels, third order 16, and fourth order 25, with higher orders generally needed for the sharpest, most resolved spatial image. In VR pipelines, ambisonic beds are commonly used for ambience and for spatialized 360 recordings, often alongside individually placed mono objects for discrete sources. The widely used interchange form is the AmbiX convention (ACN channel ordering with SN3D normalization), which platforms such as YouTube and Resonance Audio adopted.

Distance, occlusion, and room acoustics

Direction is only part of localization; a convincing scene also needs distance and environment. Distance is conveyed mainly by overall level, by the ratio of direct sound to reflected sound, and at close range by near-field effects. Meta's audio documentation states that accurate reflections are the most important feature for simulating distance, with close sources dominated by the direct signal and barely audible reflections, while distant sources carry proportionally more reverberant energy. Sources very close to the head also need near-field rendering, which approximates the acoustic diffraction around the head to better represent sounds nearer than about one meter.

Occlusion is the muffling and attenuation that happens when solid geometry sits between a source and the listener. Spatializers model this by tracing rays or paths through the scene's geometry: Steam Audio, for example, can model raycast occlusion of direct sound by solid objects, partial occlusion of non-point sources, and transmission of sound through the occluding material.

Room acoustics covers the early reflections and late reverberation produced by a space's size, shape, and surface materials, which give the ear strong cues about where the listener is and how far away a source is. Engines render this in two broad ways. Convolution reverb encodes the reflections in an impulse response, a filter that reproduces each reflection as it arrives; it is the most detailed approach but costs significant CPU. Parametric (artificial) reverb instead reduces the reflected field to a few numbers describing how energy decays over time, typically using a feedback delay network, which is cheaper. Steam Audio offers both, and can also bake and update sound propagation paths so audio travels realistically down corridors and through doorways.

Output hardware

Because spatial audio relies on each ear hearing the right signal, headphones are the natural output for VR and AR. The trade-off is that sealed on-ear or in-ear headphones bypass the listener's own ear and head filtering, can become hot or uncomfortable over long sessions, and may color the midrange and treble in ways that interfere with the subtle HRTF cues the rendering depends on.

Several headsets address this with near-ear or off-ear speakers that sit just outside the ear rather than on or in it. The Valve Index uses a pair of ultra near-field, full-range, off-ear (extra-aural) speakers built around 37.5 mm Balanced Mode Radiator (BMR) drivers developed with Tectonic, with a stated frequency response of 40 Hz to 24 kHz. Valve placed them close enough to mimic player-relative stereo headphones yet far enough that the user's own ears and head still imprint their natural coloration on the sound. BMR drivers behave like a conventional speaker at low frequencies but avoid the diaphragm break-up of ordinary drivers at high frequencies, so the sound stays consistent even when the speaker is slightly out of position, which matters for a device people put on differently every time. Standalone headsets such as the Quest line build small speakers into the strap arms near the ears, and many headsets, including the PlayStation VR2, also provide a 3.5 mm jack so users can attach their own headphones or earbuds when they want isolation or a personally tuned profile.

Spatial audio SDKs and technologies

Several software development kits and platform technologies handle the spatialization work for developers. The major ones are summarized below.

Technology	Developer	License / access	Spatialization approach	Notes
Meta XR Audio SDK (formerly Oculus Audio SDK / Oculus Spatializer)	Meta	Free, proprietary	HRTF object spatialization, first-order AmbiX ambisonics, near-field rendering, room acoustics (early reflections and reverberation)	Successor to the Oculus Spatializer plugin; targets Quest and other Meta Horizon OS devices
Steam Audio (internally Phonon)	Valve	Free, open source (Apache 2.0 since v4.5.2)	HRTF binaural rendering, geometry-based occlusion and transmission, convolution and parametric reverb, first and higher order ambisonics, baked sound propagation	Integrations for Unity, Unreal Engine, FMOD Studio, Wwise, and a C API
Resonance Audio	Google	Free, open source (Apache 2.0, March 2018)	Higher-order ambisonics with HRTFs; AmbiX (ACN/SN3D) interchange	Cross-platform: Unity, Unreal, FMOD, Wwise, Web, Android, iOS; originally Google VR Audio
Spatial Sound (Windows Sonic, plus Dolby Atmos and DTS:X for Headphones)	Microsoft	Built into Windows / Xbox; Windows Sonic is free	HRTF-based object audio; apps emit audio objects positioned in 3D space	Windows Sonic introduced in 2016; enabled by default and DSP-offloaded on HoloLens 2
Tempest 3D AudioTech	Sony	Built into PS5 / PlayStation VR2	HRTF processing accelerated by a dedicated AMD GPU compute unit; positions hundreds of sources in a 360-degree sphere	Five HRTF profiles at launch derived from over 100 people; works over headphones and TV speakers

Meta XR Audio SDK

The Meta XR Audio SDK is Meta's current spatial audio toolkit and the successor to the Oculus Spatializer plugin and Oculus Audio SDK, providing the same core capabilities under the Meta XR branding. It performs HRTF-based object spatialization, decodes first-order ambisonics in AmbiX format using spherical-harmonic decoding (which Meta says gives a flatter frequency response, better externalization, less smearing, and lower compute cost than virtual-speaker decoding), and simulates room acoustics through early reflections and late reverberation tied to a room's size, shape, and materials. Its near-field rendering approximates head diffraction for sources closer than one meter.

Steam Audio (Phonon)

Steam Audio, whose internal API carries the name Phonon, is Valve's free spatial audio solution for games and VR. It provides HRTF-based binaural rendering aimed at accurate localization with minimal frequency coloration, geometry-driven occlusion and reflection using a scene's existing geometry, sound propagation that captures how audio travels through an environment, and rendering into first or higher order ambisonics. It can spatialize point sources and ambisonic sources using any HRTF the user supplies. In February 2024, with version 4.5.2, Valve released the entire Steam Audio codebase, including the SDK and all plugins, under the Apache 2.0 license, its first fully open-source release.

Resonance Audio

Resonance Audio began at Google as Google VR Audio and was open-sourced under the Apache 2.0 license on 14 March 2018, including a standalone library, engine plugins, a VST plugin, and examples. It uses highly optimized higher-order ambisonics with HRTFs to spatialize many simultaneous sources, and it ships a reference implementation of YouTube's ambisonic decoder compatible with the AmbiX (ACN/SN3D) format. Plugins cover Unity, Unreal, FMOD, and Wwise, with support spanning the web, Android, and iOS.

Microsoft Spatial Sound

Microsoft Spatial Sound is the spatial audio platform built into Windows, Xbox, and HoloLens. Its APIs let developers create audio objects that emit sound from positions in 3D space, which are then rendered to one of several back ends: Windows Sonic for Headphones (Microsoft's own free renderer, introduced in 2016), Dolby Atmos for Headphones, or DTS:X for Headphones. Each back end applies HRTF-based filtering to place sounds around the listener. On HoloLens 2, Microsoft Spatial Sound is enabled by default and runs on a hardware DSP offload built for Windows Sonic.

Sony Tempest 3D AudioTech

Sony's Tempest 3D AudioTech is the spatial audio engine of the PlayStation 5 and is used by the PlayStation VR2. It runs on the Tempest Engine, a re-engineered AMD GPU compute unit stripped of its caches and fed by DMA transfers, which Sony dedicates to decompressing and positioning sound. It applies HRTF processing to place hundreds of sources in a 360-degree sphere around the player, and ships with at least five HRTF profiles at launch derived from testing more than 100 people, with the longer-term goal of generating personalized profiles from photos of a user's ears. On PlayStation VR2 the audio adapts dynamically to the user's position and head movements and is delivered through the headset's built-in stereo 3.5 mm headphone jack.

See also

• Immersion
• Presence
• Valve Index
• PlayStation VR2
• Field of view