Toggle search
Toggle menu
4.8K
5.5K
229
46.5K
VR & AR Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Mixed reality capture

From VR & AR Wiki
More actions

Mixed reality capture (MRC) is the recording or live streaming of video that combines real-world camera footage with computer-generated imagery from a virtual reality or augmented reality application, so that people without a headset can see what the user is doing. The term is used in two related senses. On Microsoft's HoloLens, mixed reality capture is the built-in feature that merges the headset's rendered holograms with the view of its front-facing photo/video camera, producing first-person photos, videos and live streams.[1][2] For VR headsets, the term describes a third-person technique in which an external camera films the player, usually in front of a green screen, and the footage is composited between foreground and background layers rendered from a virtual camera that matches the real camera's position, so the player appears inside the game world.[3][4]

The VR form of the technique was popularized in early 2016 by Colin and Sarah Northway, developers of Fantastic Contraption, who used it first for live streams on Twitch and then, with trailer editor Kert Gartner, for a trailer released ahead of the HTC Vive launch.[5][6] Support was later built into SteamVR's Unity plugin, the Oculus PC and Quest SDKs, Unreal Engine and the third-party tool LIV, which Meta made its official capture solution for Quest in 2024.[7][8]

Reviewed 29 September 2026. Every cited Microsoft, Meta, Epic, Steam, LIV and GitHub page, press article and paper (with DOIs) was reopened to check definitions, capture formats, dates, version numbers, study figures and quotations. About review dates.

Terminology

The term mixed reality was given a formal definition in Paul Milgram and Fumio Kishino's 1994 taxonomy, which described mixed reality displays as those that merge real and virtual worlds "somewhere along the 'virtuality continuum'" between completely real and completely virtual environments (see Reality-virtuality continuum). The paper called the augmentation of a real environment with virtual objects augmented reality, and proposed the name augmented virtuality for "completely graphic display environments" to which video "reality" is added. Among the existing mixed reality systems it listed were monitor-based video displays with overlaid computer graphics, a combination the authors noted had been technically possible for some time, "most notably by means of chroma-keying".[9]

Microsoft's HoloLens documentation mostly writes the feature name in lower case ("mixed reality capture"), while Oculus and Meta developer documentation capitalizes it as "Mixed Reality Capture".[1][4] The third-person output is also called "mixed reality video", or a "mixed reality" stream when broadcast live.[6][7]

How it works

Third-person capture for VR

A VR mixed reality capture setup has four parts: a physical camera pointed at the player, a way of tracking that camera or at least knowing its fixed pose, a virtual camera in the application that renders the scene from the same position and field of view, and a compositor that combines the layers in real time or in post-production.[3][10]

The application renders two images from the virtual camera: a background layer containing everything behind the headset and a foreground layer containing everything between the camera and the headset. The live camera image of the player is placed between them, so that virtual objects can pass in front of the player as well as behind.[5][4] In the setup Gartner used for the Fantastic Contraption trailer, the game window was split into quadrants: the foreground layer on a blue background in the upper right, the background layer in the upper left, an in-game composite of both in the lower left (used to capture correct shadows from foreground objects) and a smoothed first-person view in the lower right, captured at 2560x1440 and 60 frames per second.[10] Valve's SteamVR Unity plugin produced a similar four-quadrant output when an externalcamera.cfg file was present.[7]

The player is separated from the room by chroma keying against a green screen.[6][11] The underlying image operation is matting and alpha compositing. Thomas Porter and Tom Duff of Lucasfilm set out the case for four-channel images with a matte (alpha) channel and an algebra for compositing separately rendered elements in 1984.[12] In their 1996 SIGGRAPH paper "Blue Screen Matting", Alvy Ray Smith and James F. Blinn showed that pulling a matte from a single constant backing color (blue or, increasingly, green) is mathematically unsolvable as generally practiced, and described constraints that make it workable, including shooting the foreground against two different backings.[13]

Aligning the two worlds requires calibration. Meta's PC SDK documentation splits the external camera's parameters into intrinsics (resolution, field of view, exposure frequency and lens distortion), which rarely change, and extrinsics (relative pose, latency to the system and any attached tracked object), which change often.[3] Gartner described matching the virtual camera's field of view and position to the physical lens as "one of the most cumbersome and finicky parts of the whole experience".[10] Because a camera image reaches the computer later than the rendered frame, compositors delay the rendered layers to match; the Northway setup exposes a "Camera Delay" slider, and Meta's documentation tells developers to delay the rendered scene by the reported additional latency.[11][3] A moving camera is tracked by attaching a tracked controller to it: a third Vive controller in SteamVR, or an extra Oculus Touch controller held in an Oculus-approved 3D-printable mount on the Oculus Rift.[7][14]

Meta's PC documentation describes two composition modes. In direct composition the application itself captures the camera image, removes the green screen and combines the layers; in external composition a separate program produces the final composite, and the application delays passing its rendered scene by the camera's additional latency so that the images line up.[3]

Depth-based compositing

The layered method places the whole camera image of the player at a single depth between the foreground and background. In October 2016 Owlchemy Labs teased a method that fed video and per-pixel depth from a ZED stereo camera into Unity through a custom plugin and shader, cutting out and depth-sorting the player inside the engine instead of in external software such as OBS or Adobe After Effects. Owlchemy wrote that the pipeline had to bring 1080p video with depth data into Unity at 30 frames per second while the player could still reach 90 frames per second in the headset. Road to VR noted that the ZED does not use infrared sensors (as the Kinect does), which can interfere with VR positional tracking.[15] Later tools replaced the physical green screen with person segmentation: Reality Mixer, an open-source iOS and iPadOS app, uses ARKit person segmentation, and optionally the LiDAR sensor, to composite a Quest player without a PC or green screen.[16]

First-person capture on HoloLens

On HoloLens, mixed reality capture is a system feature that applications get without extra work. By default it combines the right eye's holographic output with the photo/video (PV) camera image, aligned at the focus point set by the running app; if no focus point is set, the stabilization plane defaults to 2 meters, and holograms at other distances can appear offset because the PV camera and the right display are physically apart. The capture blends holograms using their alpha channel and assumes premultiplied alpha, so apps must clear to transparent black rather than opaque black to avoid black edges or a black background in the recording.[1]

HoloLens 2 added an opt-in mode in which the app renders an additional view from the PV camera's own position. Microsoft states that this keeps hologram alignment accurate at all distances and avoids degrading the right-eye image in the headset.[1] The PreferredHologramPerspective setting that selects this mode defaulted to the display view until the June 2020 update (Windows Holographic, version 2004 build 19041.1106 and version 1903 build 18362.1064), after which it defaults to rendering from the camera.[17]

While capture runs, the system throttles the app to 30 Hz rendering, matching the 30 fps recording rate, and hologram content in the right eye may "sparkle" unless the app renders from the PV camera.[1] Users start captures by pressing both volume buttons (briefly for a photo, holding for three seconds for video), by voice ("Take a picture", "Start recording" on HoloLens 2 version 2004 and later) or from the Start menu. Live views can be shared over Miracast, through the Windows Device Portal or the HoloLens companion app (both requiring Developer mode), or with Dynamics 365 Remote Assist. Before Windows Holographic version 20H2, HoloLens 2 recordings were limited to five minutes; later versions calculate the limit from available storage.[2]

Default HoloLens mixed reality capture formats[2]
Device Photo Video Video frame rate Audio
HoloLens (1st gen) JPEG, 1408x792 MPEG-4, 1216x684 24 fps 48 kHz stereo
HoloLens 2 JPEG, 3904x2196 MPEG-4, 1920x1080 30 fps 48 kHz stereo

Developers can build their own capture UI with the Windows Media Capture APIs and an MRC video effect whose properties include HologramCompositionEnabled, RecordingIndicatorEnabled, video stabilization and a GlobalOpacityCoefficient (default 0.9 on HoloLens). Since the May 2021 update (Windows Holographic, version 21H1 build 20346.1002), apps can also request a stream of mixed reality frames through the Windows Screen capture API.[17]

Because this first-person output depends on the headset camera and moves with the wearer's head, Microsoft released Spectator View in February 2017: a DSLR camera with HDMI output carries a HoloLens on a bracket, a PC runs the same app as a shared experience with the headset, and the result is a stable third-person view of the wearer and the holograms. The code and hardware guidance were published on GitHub.[18]

History

According to Gartner, Colin and Sarah Northway came up with the idea "around the winter of 2015", when Breakwater Studios filmed a piece for Unity, by taping a controller from the Vive developer kit to the top of their camera to capture gameplay and live action at the same time.[10] By February 2016 the Northways were streaming Fantastic Contraption on Twitch with a webcam, a green screen and custom game code that separated the foreground and background, and they were considering a free Unity plug-in so other developers could do the same. Sarah Northway told UploadVR: "Anything that's good for VR is good for every one of us."[5] Their March 2016 how-to guide recommended an inexpensive Sony handycam with 1080p, 60 fps HDMI output and a capture card, and described in-game shortcuts for switching between first-person, third-person and mixed reality cameras.[11]

On 7 March 2016 UploadVR reported that version 1.0.8 of Valve's SteamVR Unity plugin had added SteamVR_ExternalCamera "for filming mixed reality videos", enabled by placing externalcamera.cfg in the project root and toggled by the presence of a third controller.[7] The Fantastic Contraption trailer, made by Gartner with a stabilized camera tracked by a Vive controller in a rented green screen studio, was released in early April 2016 ahead of the game's launch with the Vive.[10][19][6] Gartner also cut a mixed reality trailer for Owlchemy Labs' Job Simulator, though the limited scope of the mixed reality footage meant it showed mostly first-person gameplay; the Tilt Brush mixed reality trailer of the same period was made by MPC.[10]

Oculus added native mixed reality capture for the Rift in July 2017, with support in Unity, Unreal and the native SDK. The setup required an external USB or HDMI camera, a green screen and an extra Touch controller, and Oculus warned that it "does require numerous steps and a certain level of technical proficiency".[14] In December 2019 Oculus engineers described MRC for Quest: a PC calibration tool matches Touch controller poses in the camera image and tracking space, the Quest app renders background and foreground layers from the virtual camera and streams them over Wi-Fi, and an OBS plugin composites them with the camera feed. It shipped in the Oculus Integration package for Unity from version 1.38 (enabled through OVRManager) and in the OculusVR plugin for Unreal Engine 4.[4]

LIV, a company that says it has served XR developers since 2016, distributes a PC application on Steam, where the store page lists a release date of 12 August 2019; it films "your real body inside your favorite VR game" and also offers avatars and in-headset stream chat.[20][21] On 26 June 2024 Meta deprecated its own Mixed Reality Capture Tool and named LIV its successor under a multi-year partnership, funding LIV to expand support for Quest apps.[8][22] LIV SDK 2.0 entered beta for Unity with support for Quest hand tracking, passthrough, scene understanding, spatial anchors and occlusion, with Unreal support to follow; LIV states that both SDKs are MIT licensed and free for developers.[20] Meta's Unity documentation now says that mixed reality capture and virtual camera support "now uses LIV.tv's tooling and SDKs"; existing integrations keep working, but the MRC download links have been removed.[23]

In November 2024 LIV introduced the LIV Creator Kit, a Unity SDK that developers add to their games to give players a floating in-game camera for first- or third-person recording on a standalone Quest as well as on PC VR, with Gorilla Tag the first major title to implement it. UploadVR noted that compositing the player's real body is not part of the standalone kit.[24]

Selected implementations
Implementation Developer First reported Camera tracking Compositing
Fantastic Contraption streaming setup Northway Games February 2016[5] In-game camera calibration[11] Foreground and background layers keyed with a live green screen feed
SteamVR_ExternalCamera (Unity plugin 1.0.8) Valve March 2016[7] Third tracked controller Four-quadrant output for external compositing
Depth-based in-engine compositing Owlchemy Labs October 2016[15] Static or tracked camera ZED stereo depth sorted inside Unity
Spectator View Microsoft February 2017[18] HoloLens mounted on a DSLR Shared experience rendered on a PC
Oculus mixed reality capture for Rift Oculus July 2017[14] Extra Touch controller in a mount Direct or external composition[3]
Oculus MRC for Quest Oculus December 2019[4] PC calibration tool OBS plugin on a PC, layers streamed over Wi-Fi
Mixed Reality Capture framework Epic Games (Unreal Engine) Documented as beta for Unreal Engine 4.27[25] Standalone calibration tool (MRCalibration.exe) Green screen composite sent to the VR spectator screen
LIV SDK 2.0 LIV June 2024 (beta)[8] PC with external camera PC compositing, official Quest solution

The Unreal Engine plugin is marked beta and only supports a specific list of video capture devices; it composites but does not record, sending its output to the VR spectator screen by default.[25]

Applications

In consumer VR, mixed reality capture has mainly been used for game trailers and live streams. Kotaku's Stephen Totilo wrote that it is "notoriously difficult to convey what a virtual reality video game looks and feels like to people who aren't wearing virtual reality goggles" and that the Fantastic Contraption trailer "nearly nails it", while cautioning that it "is, of course, not exactly what the game looks like".[26] Gartner argued that seeing someone inside VR interacting with the environment makes viewers understand it "in a way that just isn't same with first person footage".[10] Microsoft lists app demos, user education, Microsoft Store screenshots, user research and problem solving as uses of HoloLens captures, and live sharing with a remote expert is part of Dynamics 365 Remote Assist.[1][2] Meta's 2025 capture guide for developers publishing Quest apps lists a movable MRC camera through LIV and Meta's fixed-position MRC camera among its third-person options, alongside alpha-channel, green screen and LiDAR depth approaches.[27]

Research

Katharina Emmerich, Andrey Krekhov, Sebastian Cmentowski and Jens Krüger compared first-person and mixed reality third-person streams in a paper at CHI 2021. They noted that many VR streamers "use a green screen and external cameras to blend themselves into the virtual world", and produced their third-person videos with LIV and a static game camera behind the player. In an online survey of 217 participants who watched Beat Saber, SUPERHOT VR and Stand Out: VR Battle Royale, 134 preferred the first-person version, 60 the third-person version and 23 had no preference. Preference depended on the game: Beat Saber viewers were nearly evenly split (31 to 27). Across all games, the reason most often given for preferring third person was being able to see the player and their movements.[28]

Researchers have also looked at moving compositing onto standalone headsets. A September 2025 arXiv preprint (not peer reviewed) by Muhammad Kaif Laghari and colleagues reported a simulation-based feasibility study of an LIV SDK-like compositing pipeline on the Meta Quest 3. Its benchmarks indicated that the headset's Qualcomm Snapdragon XR2 Gen 2 could support lightweight native compositing at 720p and 30 fps using 95 percent of available resources, and that thermal limits would restrict operation to 5 to 10 minutes before throttling.[29]

Limitations

Third-person capture needs extra hardware: a camera, usually a green screen and lighting, and on PC-based systems a capable computer.[4][14] Quest MRC relied on a Wi-Fi link between headset and PC, so connection quality affected results.[4] Reality Mixer's documentation states that mixed reality capture broke on the Quest 2, Quest 3 and Quest Pro after system version 51.[16] On HoloLens, capture halves the frame rate to 30 Hz and resolution can drop when another app uses the camera, during live streaming or when resources are low.[2]

See also

References

  1. ↑ 1.0 1.1 1.2 1.3 1.4 1.5 "Mixed reality capture overview". Microsoft Learn. Microsoft. 2022-03-18. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/mixed-reality-capture-overview. Retrieved 2026-09-29.
  2. ↑ 2.0 2.1 2.2 2.3 2.4 "Capture, record, and share mixed reality photos and videos". Microsoft Learn. Microsoft. 2024-06-27. https://learn.microsoft.com/en-us/hololens/holographic-photos-and-videos. Retrieved 2026-09-29.
  3. ↑ 3.0 3.1 3.2 3.3 3.4 3.5 "Mixed Reality Capture". Meta Horizon OS Developers. Meta Platforms. https://developers.meta.com/horizon/documentation/native/pc/dg-mrc/. Retrieved 2026-09-29.
  4. ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 Xiang Wei, Kevin Xiao (2019-12-13). "Mixed Reality Capture on Quest: Workflow, Integrations + Insights". Meta Horizon OS Developers Blog. Oculus. https://developers.meta.com/horizon/blog/mixed-reality-capture-on-quest-workflow-integrations-insights/. Retrieved 2026-09-29.
  5. ↑ 5.0 5.1 5.2 5.3 Ian Hamilton (2016-02-13). "Fantastic Contraption Devs Considering Unity Plug-in For Mixed Reality Streaming". UploadVR. https://www.uploadvr.com/northway-unity-plug-in-mixed-reality/. Retrieved 2026-09-29.
  6. ↑ 6.0 6.1 6.2 6.3 Alex Wawro (2016-04-04). "Need to make a great trailer for your VR game? Consider a green screen". Game Developer. https://www.gamedeveloper.com/design/need-to-make-a-great-trailer-for-your-vr-game-consider-a-green-screen. Retrieved 2026-09-29.
  7. ↑ 7.0 7.1 7.2 7.3 7.4 7.5 Ian Hamilton (2016-03-07). "Mixed Reality Gets Official Support With Unity Plug-In, Watch This Impressive New Video". UploadVR. https://www.uploadvr.com/mixed-reality-videos-get-official-support-with-unity-plug-in/. Retrieved 2026-09-29.
  8. ↑ 8.0 8.1 8.2 David Heaney (2024-06-26). "LIV Is Now The Official Quest Third-Person Capture Tool". UploadVR. https://www.uploadvr.com/meta-is-deprecating-quest-mrc-capture-tool-and-officially-supporting-liv-instead/. Retrieved 2026-09-29.
  9. ↑ Paul Milgram, Fumio Kishino (1994-12). "A Taxonomy of Mixed Reality Visual Displays". IEICE Transactions on Information and Systems, vol. E77-D, no. 12, pp. 1321-1329. https://www.alice.id.tue.nl/references/milgram-kishino-1994.pdf. Retrieved 2026-09-29.
  10. ↑ 10.0 10.1 10.2 10.3 10.4 10.5 10.6 Kert Gartner (2017-04-05). "Making High Quality Mixed Reality VR Trailers and Videos". Kert Gartner. http://www.kertgartner.com/making-mixed-reality-vr-trailers-and-videos. Retrieved 2026-09-29.
  11. ↑ 11.0 11.1 11.2 11.3 Colin Northway (2016-03-25). "How To Mixed Reality". Northway Games. https://northwaygames.com/how-to-mixed-reality/. Retrieved 2026-09-29.
  12. ↑ Thomas Porter, Tom Duff (1984-07). "Compositing Digital Images". Computer Graphics (Proceedings of SIGGRAPH '84), vol. 18, no. 3, pp. 253-259. doi:10.1145/800031.808606. https://keithp.com/~keithp/porterduff/p253-porter.pdf. Retrieved 2026-09-29.
  13. ↑ Alvy Ray Smith, James F. Blinn (1996-08). "Blue Screen Matting". Proceedings of SIGGRAPH '96, pp. 259-268. doi:10.1145/237170.237263. https://graphics.stanford.edu/courses/cs148-09-fall/papers/smith-blinn.pdf. Retrieved 2026-09-29.
  14. ↑ 14.0 14.1 14.2 14.3 Scott Hayden (2017-07-04). "Oculus Brings Mixed Reality Capture Support to Rift". Road to VR. https://www.roadtovr.com/oculus-brings-native-mixed-reality-capture-support-rift/. Retrieved 2026-09-29.
  15. ↑ 15.0 15.1 Scott Hayden (2016-10-04). "Owlchemy Labs Teases New In-Engine Mixed Reality Tech". Road to VR. https://www.roadtovr.com/owlchemy-labs-teases-new-engine-mixed-reality/. Retrieved 2026-09-29.
  16. ↑ 16.0 16.1 fabio914. "RealityMixer: Mixed Reality app for iOS". GitHub. https://github.com/fabio914/RealityMixer. Retrieved 2026-09-29.
  17. ↑ 17.0 17.1 "Mixed reality capture (DirectX)". Microsoft Learn. Microsoft. 2022-04-20. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/native/mixed-reality-capture-directx. Retrieved 2026-09-29.
  18. ↑ 18.0 18.1 Brandon Bray (2017-02-13). "How-to: Spectator View, a new tool to help others see what you see in HoloLens". Windows Blogs. Microsoft. https://blogs.windows.com/devices/2017/02/13/spectator-view-new-tool-help-others-see-see-hololens/. Retrieved 2026-09-29.
  19. ↑ Ian Hamilton (2016-04-03). "Watch The 'Fantastic Contraption' Trailer Sell VR To The Masses". UploadVR. https://www.uploadvr.com/fantastic-contraption-mixed-reality-trailer/. Retrieved 2026-09-29.
  20. ↑ 20.0 20.1 AJ Shewki. "LIV partners with Meta". LIV Blog. LIV Inc. https://www.liv.tv/blog/liv-partners-with-meta-to-become-the-official-capture-solution-for-meta-quest. Retrieved 2026-09-29.
  21. ↑ "LIV on Steam". Steam. Valve. https://store.steampowered.com/app/755540/LIV/. Retrieved 2026-09-29.
  22. ↑ Scott Hayden (2024-06-26). "Meta Ditches Its Own Tools in Favor of LIV Partnership for Mixed Reality Capture on Quest". Road to VR. https://roadtovr.com/meta-ditches-mixed-reality-tools-for-liv/. Retrieved 2026-09-29.
  23. ↑ "Mixed reality capture". Meta Horizon OS Developers. Meta Platforms. https://developers.meta.com/horizon/documentation/unity/unity-mrc/. Retrieved 2026-09-29.
  24. ↑ David Heaney (2024-11-01). "LIV Can Now Capture Standalone On Quest, And Gorilla Tag Will Support It". UploadVR. https://www.uploadvr.com/liv-creator-kit-standalone-quest/. Retrieved 2026-09-29.
  25. ↑ 25.0 25.1 "Mixed Reality Capture Quick Start". Unreal Engine 4.27 Documentation. Epic Games. https://dev.epicgames.com/documentation/en-us/unreal-engine/mixed-reality-capture-quick-start?application_version=4.27. Retrieved 2026-09-29.
  26. ↑ Stephen Totilo (2016-04-04). "Finally, A Great Trailer For A Virtual Reality Game". Kotaku. https://kotaku.com/finally-a-great-trailer-for-a-virtual-reality-game-1768942475. Retrieved 2026-09-29.
  27. ↑ "Capture MR and VR Apps for Publishing". Meta Horizon OS Developers. Meta Platforms. 2025-06-27. https://developers.meta.com/horizon/resources/video-capture-mr-vr/. Retrieved 2026-09-29.
  28. ↑ Katharina Emmerich, Andrey Krekhov, Sebastian Cmentowski, Jens Krüger (2021-05). "Streaming VR Games to the Broad Audience: A Comparison of the First-Person and Third-Person Perspectives". Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). doi:10.1145/3411764.3445515. https://arxiv.org/abs/2101.04449. Retrieved 2026-09-29.
  29. ↑ Muhammad Kaif Laghari, Areeb Ahmed Shaikh, Faiz Khan, Aafia Gul Siddiqui (2025-09-23). "Native Mixed Reality Compositing on Meta Quest 3: A Quantitative Feasibility Study of ARM-Based SoCs and Thermal Headroom". arXiv. https://arxiv.org/abs/2509.18929. Retrieved 2026-09-29.
Retrieved from "https://vrarwiki.com/index.php?title=Mixed_reality_capture&oldid=46110"