Microsoft Holoportation
More actions
| Microsoft Holoportation | |
|---|---|
| Information | |
| Type | Telepresence |
| Industry | 3D capture and communication research |
| Developer | Microsoft Research |
| Written In | C++, C#, CUDA (open-source 3D Telecommunications code) |
| Operating System | Windows 11 (fusion, render and viewer PCs); Linux on NVIDIA Jetson camera units (3D Telecommunications release) |
| License | MIT License (3D Telecommunications code, released December 2025) |
| Supported Devices | HoloLens, Oculus Rift DK2, HTC Vive, Azure Kinect capture pods |
| Release Date | 2016 (research demonstrations); December 2025 (open-source release as 3D Telecommunications) |
| Website | https://www.microsoft.com/en-us/research/project/holoportation-3/ |
Microsoft Holoportation (styled by Microsoft as Holoportation™) is a real-time 3D capture and telepresence technology developed by Microsoft Research. It reconstructs high-quality 3D models of people, compresses them and transmits them to remote viewers live; combined with a mixed reality headset such as HoloLens, it lets users "see, hear, and interact with remote participants in 3D as if they are actually present in the same physical space", in Microsoft's description.[1]
Microsoft demonstrated the system publicly in early 2016, at the TED conference in February and in a Microsoft Research video in March, shortly before the first HoloLens development kits shipped, and described it in the paper "Holoportation: Virtual 3D Teleportation in Real-time" at the ACM UIST symposium in October 2016.[2][3] The technology was later made mobile, rebuilt around Azure Kinect depth cameras, incorporated into Microsoft Mesh, and used for 3D telemedicine between surgeons in Scotland and patients in Ghana. In December 2025 Microsoft released the system's code as open source under the name 3D Telecommunications (3DTC).[1][4]
History
2016 research system
The project came from Microsoft Research's Interactive 3D Technologies group. Computerworld reported on 25 March 2016 that the group's capture rig mapped people's movements in real time and sent 3D renderings of them to HoloLens wearers, and that partner research manager Shahram Izadi appeared in the group's demonstration video alongside other people, including his daughter, who were recorded in another room and shown as digital renderings. The same report said that Alex Kipman, head of the HoloLens project, had shown the capability at the TED conference in Vancouver the previous month, where NASA scientist Jeff Norris joined him on stage digitally with the help of a specialized camera rig. Because Holoportation required special cameras, Computerworld wrote, it would not be immediately usable by recipients of the first HoloLens developer kits, which Microsoft was about to ship, and it was unclear if or when the technology would be commercially available.[2] New Atlas quoted Izadi describing playback of a recorded session as "almost like walking into a living memory that I can see through another pair of eyes", and reported that recordings could be shrunk to tabletop size.[5]
The technical description followed at UIST 2016 in a paper by Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello and 20 co-authors, with Izadi as the last author. The paper presented Holoportation as "an end-to-end system for augmented and virtual reality telepresence".[3][4] Its temporal reconstruction built on Fusion4D, a real-time performance capture method published by many of the same authors in ACM Transactions on Graphics in 2016.[3][6][7]
Mobile Holoportation
On 7 January 2017 Microsoft Research published a video, presented by researcher Spencer Fowers, titled "Mobile Holoportation", a short demonstration of the team's efforts to make Holoportation "truly mobile".[8] According to the project FAQ, a new compression approach cut the bandwidth needed by 97 percent, to 30-50 Mbps, which allowed the system to run in a moving vehicle within Wi-Fi range. The FAQ says the system needs only two depth-capable cameras, although more cameras give a better 3D model, and that the in-car scenario was meant for a back-seat passenger with no interaction with the driver. The other engineering problems it lists were changing lighting and backgrounds, vibration, and fitting the compute into a vehicle. Remote participants could be viewed with a head-mounted display "like the Hololens or HTC Vive".[9]
A later version of the system uses Azure Kinect sensors for depth and color capture; the project page states that the team found the Azure Kinect gave "some of the highest quality depth data of any RGB+D sensor on the market".[1]
Microsoft Mesh
Microsoft announced Mesh at its Ignite conference on 2 March 2021. Kipman appeared on the Ignite virtual stage "as a fully realized holoportation of himself", narrating the show's opening experience, and Microsoft said that people would first appear in Mesh as avatars and "over time use holoportation to project themselves as their most lifelike, photorealistic selves".[10] Microsoft's technical overview of Mesh described photorealistic 360-degree holoportation using outside-in sensors, where the sensors could be a custom rig such as a Mixed Reality Capture Studio or Azure Kinect cameras capturing depth images.[11] The Holoportation project page states that the technology "has been incorporated into Microsoft Mesh".[1] On 1 December 2025 Microsoft retired the Mesh PC and Quest apps, Mesh on the web and the Immersive space (3D) view in Teams meetings, replacing them with immersive events in Microsoft Teams.[12]
3D telemedicine
In December 2019 Microsoft researchers began discussing a 3D telemedicine system based on Holoportation with the Canniesburn Plastic Surgery Unit in Glasgow and Korle Bu Teaching Hospital in Accra, Ghana; Microsoft says the collaboration began with a speculative email to its Special Projects team in late 2019.[13][4] Development began at Canniesburn in March 2020 and was fast-tracked as a COVID-19 research project for remote consultations.[14][15] The research team visited Korle Bu in February 2022 and installed what Microsoft calls the first known 3D telemedicine system in Africa, a portable version with improved lighting and cameras; doctors in Glasgow used it for pre-operative and post-operative multidisciplinary team meetings with patients in Accra.[13]
The system later became a mobile unit that fits in the back of a van. Fowers said the original system "required up to three gigabytes of bandwidth" and needed fiber-optic links for demonstrations, while the van version ran over 4G cellular networks from Ghana to Brazil, Rwanda and Glasgow at the same time. By December 2025 doctors in Ghana had seen 30 patients with the system, and a second unit had been donated to the Ghanaian IT provider ECL Global.[4]
Open-source release
On 17 December 2025 Microsoft Research announced that it was releasing the 3D Telecommunications technology under an open-source license so that outside researchers and organizations could use and modify it. Fowers is a principal researcher on the Special Projects team at Microsoft Research, which Microsoft says "launched and nurtured" Holoportation from a research project through multiple 3DTC deployments. Fowers said the team had "pushed that technology about as far as we can go" and was "ready to hand it off".[4] The code is published on GitHub as microsoft/3DTelecommunications under the MIT License.[7]
How it works
Original 2016 system
The UIST 2016 system placed eight camera pods around the edge of a room, pointing inward. Each pod held two near-infrared cameras and one color camera on an optical bench, plus a laser and diffractive optical element that projected a pseudo-random pattern, using the same design as the first Kinect. The rig used 24 PointGrey Grasshopper cameras with 4-megapixel resolution, synchronized by an external trigger at 30 fps.[3] Depth came from active stereo matching rather than structured light or time-of-flight sensing; the authors wrote that structured light suffered from interference between devices and that time-of-flight had multi-path problems. With a 15 cm baseline, the paper reports an average depth error of 3 mm at 1 m and 6 mm at 1.5 m.[3]
At each capture site, four PCs (each with two NVIDIA Titan X GPUs and handling two pods) computed depth maps and foreground segmentation, then sent them over point-to-point 10 Gbps links to a dual-GPU fusion machine. That machine fused the depth maps into a single deforming mesh in an average of 29 ms per frame (about 34 fps). The authors split the non-rigid reconstruction across two GPUs: one estimated frame-to-frame motion and the other refined it and performed volumetric fusion.[3]
For transmission, mesh positions and normals were reduced to 16-bit floats and the index data compressed with LZ4; the color images from all eight color cameras were also LZ4-compressed after background regions were set to a constant color using the segmentation masks. The paper reports an average transfer rate of 1-2 Gbps for a 30 fps capture stream, with under 10 ms added by compression.[3] For untethered headsets like HoloLens, rendering was offloaded to a desktop PC on the receiving side, which predicted the headset pose, rendered the view and streamed it as video to the headset. Rendering one eye took 6 ms on a desktop GPU, enough to display at 60 Hz on HoloLens and 75 Hz on the Oculus Rift DK2.[3]
The system used spatial audio. Each headset wearer's microphone audio (mono, 11 kHz, 16-bit PCM) was sent in 20 ms chunks together with the wearer's head pose, and the receiving side placed the voice at the remote person's position using a head-related transfer function through the XAudio2 framework.[3]
Interaction features
The paper describes one-to-one sessions between two capture rigs, in which objects and furniture in each room are captured along with the people, and one-to-many live broadcasts. Sessions could be recorded and replayed from any viewpoint and at any scale, including room-sized events scaled down to fit on a coffee table, with pause, rewind and fast-forward.[3] The authors also showed Holoportation inserting a user's own captured body into a virtual reality scene, and a proof-of-concept "headset removal" prototype that used small cameras on the rim of the headset to project images of the eye region onto a 3D model of the wearer's face.[3]
3D Telecommunications system
The telemedicine version uses ten capture devices placed around the patient, each pairing an Azure Kinect sensor with an NVIDIA Jetson Nano computer. A GPU workstation fuses the depth maps into a streaming 3D model, a second desktop colorizes it, and a viewer application lets the clinician inspect the model and draw on it in 3D; the patient side joins through a Microsoft Teams call, where the model view can be screen-shared.[15] The GitHub repository describes the project as "utilizing Holoportation technology to provide live volumetric capture", a live form of volumetric video, and splits the system into the components below.[7]
| Component | Role | Platform |
|---|---|---|
| Fusion | Builds a watertight volumetric mesh in real time from any number of RGB-D streams; based on Fusion4D | C++ and CUDA on Windows 11, NVIDIA RTX 2080 Ti or larger, 10 Gb Ethernet |
| Render | Textures the mesh with the camera images and streams the model to viewers | C# Unity project (Unity 2019.4.14f1) |
| Viewer | Displays the textured 3D model and the 2D camera views, with drawing and capture tools | C# Unity project |
| Control Panel | Runs calibration and live sessions, distributes configuration and monitors all components | C# .NET with ZeroMQ |
| K4AToFusion | Pulls images from an Azure Kinect and sends them to the Fusion server | C++ on NVIDIA Jetson Nano (Linux) |
| Calibration software | Computes world coordinates of all cameras from synchronized recordings | C++ |
Microsoft says the current 3DTC system costs about 90 percent less than the original research setup, runs on two standard computers with off-the-shelf imaging components, and needs 97 percent less bandwidth without loss of quality.[4]
Evaluation
The UIST paper included a preliminary user study with 10 participants who carried out a social "tell-a-lie" task and a block-arranging task in both AR and VR conditions. 70 percent of participants agreed or strongly agreed that their partner looked like a real person. Participants said AR felt like the partner had come into their room, while VR felt like being in the partner's space; a few reported discomfort from latency in VR that was not seen in AR, and the headset itself blocked eye contact in both conditions.[3] The authors listed the hardware requirements, the 10 Gigabit Ethernet link between rooms, color artifacts under heavy occlusion and reconstruction errors on small geometry such as fingers as limitations.[3]
The telemedicine work was evaluated in three studies during the trial in Scotland: a clinician feedback study (23 clinicians, November-December 2020), a patient feedback study (26 patients, July-October 2021) and a safety and reliability cohort (40 patients, October 2021-March 2022). The authors reported better satisfaction (p<0.0001), presence (p<0.0001) and quality (Telehealth Usability Questionnaire, p=0.0002) for 3D than for 2D telemedicine, and 95 percent clinical concordance with face-to-face consultations.[14][13] Microsoft's summary gives patient satisfaction of 88 percent for 3D against 51 percent for 2D, and realism of 80 percent against 53 percent.[13] The paper received the Top Paper award at BAPRAS 2024, and Microsoft says a randomized controlled trial is under way.[15]
NASA space station demonstration
NASA used the word "holoportation" for an 8 October 2021 demonstration in which flight surgeon Josef Schmid and AEXA Aerospace chief executive Fernando De La Pena Llaca were "holoported" to the International Space Station to talk with ESA astronaut Thomas Pesquet. NASA's account says the demonstration used "the Microsoft Hololens Kinect camera and a personal computer with custom software from Aexa", and that holoportation "has been in use since at least 2016 by Microsoft".[16] NASA's account credits the software to AEXA; it does not say that the Microsoft Research system itself was used.
Current status
As of October 2026 the Holoportation code is public as the 3D Telecommunications repository, which was last updated in August 2026.[7] Korle Bu Teaching Hospital continues to use the system to train reconstructive surgeons, ECL Global has agreed to support and develop it further, and NHS Scotland doctors plan to keep using it for international multidisciplinary consultations.[4] The standalone Microsoft Mesh apps, part of the platform that incorporated the technology, were retired in December 2025 in favor of immersive events in Teams.[12]
See also
References
- ↑ 1.0 1.1 1.2 1.3 "Holoportation". Microsoft Research. Microsoft. https://www.microsoft.com/en-us/research/project/holoportation-3/. Retrieved 2026-10-06.
- ↑ 2.0 2.1 Blair Frank (2016-03-25). "Microsoft wants to 'holoport' you into a conversation elsewhere". Computerworld. https://www.computerworld.com/article/3048371/microsoft-wants-to-holoport-you-into-a-conversation-elsewhere.html. Retrieved 2026-10-06.
- ↑ 3.00 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09 3.10 3.11 3.12 Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, et al. (2016-10-16). "Holoportation: Virtual 3D Teleportation in Real-time". Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), ACM, pp. 741-754. doi:10.1145/2984511.2984517. https://doi.org/10.1145/2984511.2984517. Retrieved 2026-10-06.
- ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 "3D Telecommunications goes open source". Microsoft Research. Microsoft. 2025-12-17. https://www.microsoft.com/en-us/research/story/3d-telecommunications-goes-open-source/. Retrieved 2026-10-06.
- ↑ Eric Mack (2016-03-27). "Microsoft's "holoportation" lets you augment someone else's reality". New Atlas. https://newatlas.com/microsoft-holoportation-hololens-virtual-reality-hologram/42501/. Retrieved 2026-10-06.
- ↑ Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, et al. (2016-07-11). "Fusion4D: Real-time Performance Capture of Challenging Scenes". ACM Transactions on Graphics, vol. 35, no. 4. doi:10.1145/2897824.2925969. https://doi.org/10.1145/2897824.2925969. Retrieved 2026-10-06.
- ↑ 7.0 7.1 7.2 7.3 "microsoft/3DTelecommunications". GitHub. Microsoft. https://github.com/microsoft/3DTelecommunications. Retrieved 2026-10-06.
- ↑ Spencer Fowers (2017-01-07). "Mobile Holoportation". Microsoft Research. Microsoft. https://www.microsoft.com/en-us/research/video/mobile-holoportation/. Retrieved 2026-10-06.
- ↑ "Holoportation: FAQ". Microsoft Research. Microsoft. https://www.microsoft.com/en-us/research/project/holoportation-3/faq/. Retrieved 2026-10-06.
- ↑ Jennifer Langston (2021-03-02). ""You can actually feel like you're in the same place": Microsoft Mesh powers shared experiences in mixed reality". Microsoft Source. Microsoft. https://news.microsoft.com/source/features/innovation/microsoft-mesh/. Retrieved 2026-10-06.
- ↑ Nishant Thacker (2021-03-02). "Microsoft Mesh - A Technical Overview". Microsoft Tech Community, Mixed Reality Blog. Microsoft. https://techcommunity.microsoft.com/t5/mixed-reality-blog/microsoft-mesh-a-technical-overview/ba-p/2176004. Retrieved 2026-10-06.
- ↑ 12.0 12.1 "Microsoft Mesh". Microsoft Adoption. Microsoft. https://adoption.microsoft.com/en-us/mesh/. Retrieved 2026-10-06.
- ↑ 13.0 13.1 13.2 13.3 Chris Stetkiewicz (2023-05-30). "3D telemedicine brings better care to underserved and rural communities, even across continents". Microsoft Research Blog. Microsoft. https://www.microsoft.com/en-us/research/blog/3d-telemedicine-brings-better-care-to-underserved-and-rural-communities-even-across-continents/. Retrieved 2026-10-06.
- ↑ 14.0 14.1 Steven Lo, Spencer Fowers, Kwame Darko, Thiago Spina, et al. (2023-12). "Participatory development of a 3D telemedicine system during COVID: The future of remote consultations". Journal of Plastic, Reconstructive & Aesthetic Surgery, vol. 87, pp. 479-490. doi:10.1016/j.bjps.2022.10.012. https://doi.org/10.1016/j.bjps.2022.10.012. Retrieved 2026-10-06.
- ↑ 15.0 15.1 15.2 "3D Telemedicine". Microsoft Research. Microsoft. https://www.microsoft.com/en-us/research/project/3d-telemedicine/. Retrieved 2026-10-06.
- ↑ "Innovative 3D Telemedicine to Help Keep Astronauts Healthy". NASA. 2022-04-08. https://www.nasa.gov/humans-in-space/innovative-3d-telemedicine-to-help-keep-astronauts-healthy/. Retrieved 2026-10-06.