Holographic Processing Unit

Holographic Processing Unit
Information
Type	Sensor coprocessor
Subtype	Mixed reality headset coprocessor
Creator	Microsoft
Developer	Microsoft
Manufacturer	TSMC
Devices	Microsoft HoloLens (HPU 1.0), HoloLens 2 (HPU 2.0)
Release Date	Announced January 21, 2015; first shipped to developers in March 2016
Website	https://learn.microsoft.com/en-us/hololens/hololens2-hardware

The Holographic Processing Unit (HPU) is a custom coprocessor designed by Microsoft for its HoloLens mixed reality headsets. It sits beside the headset's main processor and handles the data from the on-board sensors, so that tracking, environment sensing and gesture input do not have to run on the general-purpose CPU. Microsoft built two generations. HPU 1.0 is in the original Microsoft HoloLens, paired with an Intel Atom "Cherry Trail" processor. The second-generation HPU 2.0 is in HoloLens 2 alongside a Qualcomm Snapdragon 850, and adds a dedicated deep neural network (DNN) core.

Both chips are built from Tensilica configurable DSP cores (a Cadence Design Systems processor line) extended with Microsoft's own instructions and fixed-function accelerators. Microsoft described them in detail at the Hot Chips symposium in 2016 and 2019.

Reviewed 4 October 2026. HPU 1.0 and HPU 2.0 specifications, Hot Chips 2016/2019 disclosures, CVPR 2017 AI coprocessor announcement, HoloLens processor specs and support dates against all 18 cited sources. About review dates.

History

Announcement (2015)

Microsoft unveiled HoloLens at its Windows 10 event on January 21, 2015. A Microsoft developer blog summary of the event listed the headset's "high-end CPU, GPU, and HPU (Holographic Processing Unit)" and said the device needed no wires and no phone or PC connection. Road to VR described the HPU as "a dedicated chip to handle sensing and interpretation of the real world as revealed by on-board sensors". Developer shipments of the headset began in March 2016.

Hot Chips 2016 disclosure

Microsoft disclosed the HPU's design at the Hot Chips conference in Cupertino, California, in August 2016, in a presentation by Microsoft Devices Group engineer Nick Baker. According to Network World, Microsoft had evaluated SoC CPUs and GPUs, traditional CPU arrays and commercial image signal processors before choosing the Cadence Tensilica design. Its priorities were support for multiple algorithms, room for future growth (cores were limited to 50% utilization) and guaranteed latency. Tom's Hardware described the result as a hybrid that uses programmable units wherever possible and fixed-function blocks where they made sense.

At the Embedded Vision Summit in 2017, HoloLens director of science Marc Pollefeys described the HoloLens hardware, including the HPU, in the opening keynote; Cadence's Paul McLellan summarized the talk.

AI coprocessor announcement (2017)

In July 2017, Harry Shum, executive vice president of Microsoft's Artificial Intelligence and Research Group, announced in a keynote at the CVPR 2017 computer vision conference that the second version of the HPU, then under development, would incorporate an AI coprocessor to "natively and flexibly implement DNNs". Pollefeys wrote that the chip supports "a wide variety of layer types, fully programmable by us", that Shum showed "an early spin of the second version of the HPU running live code implementing hand segmentation", and that the coprocessor was designed to run continuously off the next HoloLens's battery. Pollefeys explained the reason for custom silicon: any low-latency compute, such as hand tracking, "has to run off the same battery that powers everything else". Road to VR reported that the demonstration ran hand-tracking models on a first-generation HoloLens connected to a PC fitted with the new coprocessor.

HoloLens 2 and Hot Chips 2019

Microsoft announced HoloLens 2 on February 24, 2019. Road to VR's specification report described the headset as using a Snapdragon 850 "along with Microsoft's own 2nd generation 'Holographic Processing Unit', a co-processor which handles sensor input". Microsoft engineers described the headset's custom sensors and "low power compute blocks aggregated in a custom ASIC" in a paper at the 2019 Symposium on VLSI Circuits.

Elene Terry of Microsoft described the silicon inside HoloLens 2 in "The Silicon at the Heart of the Hololens 2", the final talk of Hot Chips 31, on August 20, 2019. The Register reported Terry's explanation that the relatively large 79 mm2 die lets enough heat escape, which matters because the chip sits over the wearer's forehead between the eyes.

Current status

In October 2024 Microsoft confirmed that HoloLens 2 production had ended and that there was no replacement headset. HoloLens 2 receives fixes for critical security issues and software regressions until December 31, 2027, and software support for the original HoloLens ended after December 10, 2024.

Features

Role in the headset

Pollefeys wrote that the HPU processes the information from all of HoloLens's on-board sensors, including Microsoft's custom time-of-flight depth sensor, the head-tracking cameras, the inertial measurement unit (IMU) and the infrared camera. Network World reported that it handles environmental data, aggregates sensor data and processes the wearer's gesture movements. A 2020 paper by Microsoft researchers on the HoloLens 2 Research Mode states that the HPU 2.0 runs all of the on-device computer vision algorithms, including head tracking, hand tracking, eye gaze tracking and spatial mapping, and hosts the DNN core, and that the CPU of the Snapdragon 850 "remains fully available for applications". The HPU sits at the front of the headset near the sensors, and the SoC is at the rear.

The HPU does not render application graphics. At Hot Chips 2019, Microsoft explained that the application processor runs the app and renders images, which are sent back to the HPU over MIPI. The HPU then adjusts the rendered image and sends it to the displays. Microsoft's developer documentation describes HoloLens's hologram stabilization through reprojection as a "hardware-assisted" technique. For hologram stability on HPU 2.0, Microsoft cited multiple pose updates per frame and a hardened block that decouples the render resolution from the display resolution. The HPU also timestamps sensor data as it arrives.

HPU 1.0

The first HPU is made by TSMC on a 28 nm high-performance compact (HPC) process and contains about 65 million logic gates. Most of the die is taken up by 12 compute nodes, each with two Tensilica DSP cores, for 24 cores in total. Microsoft started from Cadence's Xtensa base instruction set and added about 300 of its own extensions, according to Tom's Hardware. Fixed-function accelerators handle tasks the DSP cores cannot address satisfactorily; some stand alone and others are tightly coupled to the cores. Microsoft said the combination runs algorithms up to 200 times faster than software alone. The Register put the chip's throughput at a trillion calculations per second, and Tom's Hardware at about one trillion pixel operations per second.

The chip has 8 MB of SRAM and comes in a 12 mm by 12 mm BGA package (0.4 mm pitch, according to Cadence), and it is paired with 1 GB of low-power DDR3 (LPDDR3) memory. The Register and Cadence both reported that it dissipates less than 10 W. Tom's Hardware reported the I/O layout presented at Hot Chips: MIPI CSI inputs from the depth camera and the four environment understanding cameras, MIPI DSI outputs to the two displays, an SPI link to the IMU, I2C for a temperature sensor, and PCI Express to the Intel SoC. The host SoC is an Intel Atom x86 Cherry Trail chip running 32-bit Windows 10. Microsoft lists the headset's processors as "Intel 32-bit architecture with TPM 2.0 support" and a "Custom-built Microsoft Holographic Processing Unit (HPU 1.0)", with 2 GB of RAM in total. Cadence's summary splits that RAM as 1 GB for the main CPU and 1 GB for the HPU.

HPU 2.0

HPU 2.0 is a 79 mm2 die made on TSMC's 16FF+ process, with 123 million gates, about 2 billion transistors and 125 Mb of SRAM. It taped out in September 2016 and worked on its first silicon, according to Cadence's account of the talk. The chip runs only Microsoft workloads under a single Microsoft real-time operating system, with no memory management units and simple interrupts.

Its programmable compute is split into 13 statically assigned compute nodes: seven SIMD fixed-point (SFP) nodes for 2D processing and six vector floating-point (VFP) nodes for 3D work. Each node has two Tensilica processors, giving 26 in total, and both node types are 128 bits wide (the SFP nodes in HPU 1.0 were 64-bit). Microsoft quoted more than 1 TOPS (tera operations per second) of programmable compute. The cores have hundreds of custom instructions. Microsoft profiled its algorithms to collapse sequences of tens of operations into single instructions; its example, boxavg_2x16x8, averages a block of pixels in one cycle and saves more than 10,000 cycles per frame.

Some functions are hardened into fixed logic. A joint bilateral filter for the time-of-flight depth sensor was converted from C code into RTL, which cut its power to a third and its latency to a thirtieth. The chip also has a hardened neural network block. Terry's list of hardware blocks on the chip was: the SFP and VFP nodes, an LSR engine, DMA, monochrome ISPs, FFT for audio, the DNN, the joint bilateral filter, an interrupt controller, sensor frame timers, a DRAM scheduler (memory controller) and a classifier.

Functions handled by HPU 2.0, from a partial list given in the talk, include fully articulated hand tracking, spatial audio, 6DoF tracking, hologram stability, eye tracking, spatial reconstruction and mapping, and scene understanding such as finding floors, walls and ceilings.

Because the chip is mounted at the brow, its thermal budget is 2.4 W averaged over 30 minutes at 85 degrees Celsius. Microsoft used clock gating and power gating, runs most digital logic at 250 MHz and the compute nodes at 500 MHz, and lowered the supply voltage below 630 mV, which saved a further 20% of power. The HPU at the front of the headset talks to the application processor at the back over a PCIe 2.0 x1 link at about 100 MB/s, and has its own custom DRAM scheduler.

Comparison of generations

Feature	HPU 1.0	HPU 2.0
Headset	Microsoft HoloLens (2016)	HoloLens 2 (2019)
Host processor	Intel Atom x86 Cherry Trail	Qualcomm Snapdragon 850
Process	TSMC 28 nm HPC	TSMC 16FF+
Logic	About 65 million gates	123 million gates, about 2 billion transistors, 79 mm2 die
On-chip SRAM	8 MB	125 Mb
Programmable cores	24 Tensilica DSP cores in 12 compute nodes	26 Tensilica processors in 13 compute nodes (7 SFP, 6 VFP)
Stated compute	About one trillion calculations per second; up to 200x faster than software	More than 1 TOPS of programmable compute
Neural network hardware	None disclosed; Microsoft announced an AI coprocessor for the second version in 2017	Hardened DNN core
Power	Less than 10 W	2.4 W thermal budget averaged over 30 minutes

Microsoft compared the two at Hot Chips 2019. HPU 2.0 supports twice the field of view, has 1.7 times the compute and twice the effective DRAM bandwidth, and gives better hologram stability. It also runs four workloads the first chip did not: eye tracking, fully articulated hand tracking, semantic labeling and spatial audio.

Supported devices

Device	HPU version	Released	Status (October 2026)
Microsoft HoloLens	HPU 1.0	Developer shipments from March 2016	Software support ended December 10, 2024
HoloLens 2	HPU 2.0	Announced February 24, 2019	Production ended 2024; security and regression fixes until December 31, 2027

See also

• Apple R1
• Vision processing unit
• Movidius Myriad 2
• Windows Holographic