Immersive captions

Immersive captions present speech and relevant non-speech sound as text or visual information within virtual reality, augmented reality and 360-degree video. They are part of Accessibility in XR: a viewer can turn away from a speaker, so caption design must address where sounds originate as well as what they mean.

Reviewed 21 October 2026. W3C community guidance, original captioning studies and ImAc implementation documents About review dates.

Placement and movement

Caption position can follow the viewer or refer to the surrounding scene. These choices affect whether text stays inside the current field of view when the viewer looks around.

Placement	Behavior
Head-locked	Text keeps its position relative to the viewer's view as the head turns.
World-locked	Text occupies a location in the environment and can leave the view when the viewer turns away.
Speaker- or source-locked	Text appears beside the person or object associated with the sound.
Appear-locked	A new caption starts near the current viewing direction, then stays in that scene location until it is replaced.
Following with lag	Text returns toward a position in the viewer's view after head movement, using an animated transition or a jump. The exact movement rules depend on the implementation.

Speaker identity and sound direction

Immersive captions can identify a speaker and indicate sounds outside the visible scene. The W3C community report includes sound effects, ambience and music in its proposed coverage, alongside dialogue. It recommends alternatives for identifying speakers, such as names or icons, and cautions that color alone is insufficient. Identification should also respect narrative timing, rather than disclose a character's identity before the story does.

The Immersive Accessibility (ImAc) project implemented arrows beside subtitles that point toward an off-screen speaker and disappear when the speaker enters view. Its alternative radar display shows the viewer's orientation and the horizontal direction of the speaker associated with the active subtitle. These indicators let caption position and sound-source position remain separate: a viewer can read text in front of them while receiving a cue to look elsewhere. ImAc also experimented with captions beside speakers and sound-event emojis, but its documentation distinguishes these experiments from features included in its player version.

Appearance and controls

Caption readability depends on the changing scene behind the text. ImAc's player offered a choice of outlined text or a translucent background box and top or bottom placement. Its editor allowed caption authors to enter the additional 360-degree metadata along with ordinary subtitle properties.

A Spanish ImAc focus group requested a background box for contrast and directional information close to the dialogue, rather than elsewhere in the view. Participants differed over whether non-speech information should use text or icons and which details they wanted to receive. Such differences support configurable presentation rather than assuming that every deaf or hard-of-hearing viewer has the same preferences.

Authoring and rendering

A spatial captioning system needs more than a timed sequence of words. ImAc's 2020 technical white paper separates the placement of the subtitle region from the location of its associated audio source. It proposes carrying source direction in additional IMSC timed-text attributes, using azimuth and elevation, while separately defining how a subtitle plane maps into a 360-degree scene. The proposal uses timed-text segments to associate each caption with its direction data. These are documented ImAc extensions and proposals; their description does not establish that every IMSC player supports spatial captions.

Chris J. Hughes's framework, published online in 2021, used Three.js to compare caption positions and guide designs. Separate scene-graph groups held captions attached to the view and captions fixed in the world. Later additions supported several captions at once, stacking to reduce overlap and adjustable removal times. Its experimental JSON files stored character tracks so captions or guides could follow a moving speaker. The tracks were entered with manually defined keyframes and interpolation. This extends a single location recorded at the beginning of a caption, which can become inaccurate while a speaker moves.

Live captions can draw on automatic speech recognition. Gallaudet University's Quest 2 prototype used Azure transcription with a modified Mozilla Hubs client; this was a research implementation.

Research and limitations

The 2019 ImAc focus-group paper reports a discussion held on 28 November 2017 with 14 participants: ten deaf or hard-of-hearing home users and four professionals. Examples were shown on a flat screen, and questions included hypothetical uses of a future platform. The results identified requirements for subsequent development, rather than measuring the performance of a finished headset system. The authors also limited generalization beyond the Spanish audience studied.

The Gallaudet study, published in 2023, involved eight deaf or hard-of-hearing participants with little or no previous VR experience. Each experienced three live presentation conditions with different caption movement. Preferences varied; the study supports customization rather than a universal winning layout. It examined a single-speaker presentation task, so it does not establish results for simultaneous conversation or all prerecorded content.

A study by Marta Brescia-Zapata and colleagues, published online in November 2023, recruited 73 volunteers across Spain, the United Kingdom and Poland. It compared head-locked and scene-fixed positions between participants, while each participant saw both monochrome and colored subtitles in two short 360-degree videos. Head-locked captions reduced reported effort, but subtitle position did not significantly improve content comprehension. Speaker-specific colors improved recognition and comprehension while also increasing reported effort. The study's purpose-recorded videos had no overlapping speakers; these results therefore need care when applied to multi-speaker conversation or other content. It measured eye movements as well as questionnaires and comprehension, rather than treating stated preference as a complete measure of accessibility.

Hughes's framework was a tool for developing and testing alternatives, including potentially unsuitable combinations. Its available options do not constitute a validated best layout.

Community guidance

The W3C Immersive Captions Community Group published its final report on 12 April 2024. It is a Community Group report, outside the W3C Standards Track. Along with customizable placement and appearance, it proposes retaining multiple captions, navigating recorded material by caption and providing a separate transcript. These are recommendations and prototype ideas, rather than evidence that all headsets or media players implement them.