Lytro Immerge

Lytro Immerge
Basic Info
VR/AR	Virtual Reality
Type	Camera
Subtype	Virtual Reality Camera
Developer	Lytro
Announcement Date	November 2015
Release Date	Prototype footage first shown in August 2016; production use from 2017
Website	lytro immerge
Versions	Immerge (original spherical design, reworked into a planar rig by early 2017); Immerge 2.0 (November 2017)
Audio
Camera	Immerge 2.0: 95 cameras in alternating off-axis rows, 120-degree field of view

Lytro Immerge is a light field VR video camera system from Lytro, the company behind the Lytro ILLUM and the Original Lytro Light Field Camera. Lytro announced it in November 2015 and called it the "world's first professional light field solution for cinematic VR". Instead of recording a flat 360-degree image from one point, Immerge captured light field data over a small volume, so that a viewer wearing a headset could move their head inside that volume with six degrees of freedom (6DoF) and see correct parallax. The system came with a dedicated server that stored and processed the data captured by the camera, light field editing tools that worked with existing visual effects and editing software, and a playback engine for VR headsets.

Lytro first said prototypes would reach content creators in the first quarter of 2016, but the first footage from the camera was not shown until August 2016. The original spherical design was replaced by a planar, front-facing rig that rotated to film a full 360-degree scene, and an updated Immerge 2.0 followed in November 2017. Its productions included Hallelujah (2017), created with the VR studio Within. Lytro announced in March 2018 that it would stop taking on new productions and wind down the company.

Reviewed 13 October 2026. Checked every claim, quote, date, table cell and infobox value against the 22 cited sources, and corrected dates, attributions and the Google sale report. About review dates.

Announcement and release

Lytro raised 50 million US dollars in 2015 to develop Immerge, part of a move away from consumer light field cameras toward professional VR capture. The system was announced in early November 2015; TechCrunch, Engadget and Road to VR covered it on 5 November. Chief executive Jason Rosenthal told TechCrunch that Lytro would keep supporting and selling its older cameras, but that VR was the future of the company. Lytro said the output would work with headsets from Oculus, HTC and Sony.

Lytro did not publish a price. Rosenthal told Engadget that buying the system would likely cost "hundreds of thousands of dollars" and that renting it would probably cost a few thousand dollars a day. Road to VR reported that the equipment would be offered on a rental basis, and named Felix & Paul, Vrse (later renamed Within) and WEVR as early-access content partners. Vrse co-founder Aaron Koblin told UploadVR: "Light field technology is probably going to be at the core of most narrative VR."

The planned first-quarter 2016 date slipped. On 30 August 2016 Road to VR reported that Lytro had shown the first footage captured with the camera, which the company called the "first piece of 6DOF 360 live action VR content ever produced", and that Lytro was taking applications for prototype access. The footage was a short piece called Moon, in which an astronaut steps out of a lunar lander before a voice calls "cut" and the scene is revealed to be a film set, a reference to the moon landing hoax myth.

In February 2017 Lytro raised a further 60 million US dollars in a Series D round led by Blue Pool Capital, with EDBI, Foxconn, Huayi Brothers and Barry Sternlicht taking part. The round was meant to support Immerge and content made with it. At the same time Lytro announced a partnership with Within, the studio co-founded by Chris Milk, whose first production with the camera had finished shooting.

Design and hardware

Original spherical design (2015)

As announced, Immerge was a globe made of five rings of light field cameras and sensors. Engadget described five layers, each packed with a ring of 360-degree cameras and sensors, and reported Rosenthal as saying the design allowed accurate horizontal and vertical parallax, which is needed for computer-generated elements to sit convincingly in live-action footage. UploadVR reported that the rings held "hundreds" of individual cameras, each fitted with Lytro's own image sensor. Lytro said it had designed the sensors and the system from scratch.

Lytro said the camera captured a "light field volume"; TechCrunch explained that it recorded light data from all directions at a given location. Because the data records the direction of incoming light as well as its color, the playback software could produce a view from any position inside the captured volume, which made positional tracking possible in the headset. This is the difference between 6DoF video and ordinary 360-degree video, where the viewer can only turn their head. Six degrees of freedom combine three axes of rotation (pitch, yaw and roll) with three axes of movement (forward and back, up and down, left and right).

The rings were modular. Road to VR reported that each horizontal sensor "slice" could be removed to reduce weight and cost, at the price of lower fidelity and a smaller capture volume. UploadVR wrote that more rings and denser sensors improved positional fidelity and headset resolution, which Lytro put at "up to 2k per eye", and that the camera supported six degrees of freedom within a volume of about one meter. Road to VR described the capture area as a volume about half a meter across. Rosenthal told Engadget that "one of the layers represents somewhere between three and four times" the data and resolution of any VR camera then available.

According to Lytro, the same footage could be rendered as 180-degree or 360-degree video, in mono or stereo, or as a full light field with positional tracking, and a monoscopic feed could be streamed to a live monitor on set.

Server

Immerge included a dedicated server for storage and processing, with what Lytro called ultra-high bandwidth direct-to-disk capture. Each server held approximately an hour of 360-degree light field video. Road to VR described it as a fibre-attached storage array with enough processing power to preview renders, connected by PCI-E over fibre so that it could sit around 150 metres from the camera. Lytro's chief product officer, Ariel Braunstein, told UploadVR: "The camera is useless without the server." Road to VR reported that directors would control the system from tablet software that communicated with the server wirelessly.

Planar prototype and rotating capture (2016-2017)

The camera Lytro actually used on set was not the sphere it had announced. UploadVR's August 2016 hands-on described the prototype as 60 cameras arranged in a staggered hexagonal "wedge" about 0.75 m high and 0.75 m wide. The rig was shot from five rotational positions, giving roughly 300 camera viewpoints for a full 360-degree scene, and the viewer could move within a sphere about half a meter in diameter. By February 2017 Lytro had confirmed that it had dropped the design that captured every direction at once in favor of a planar, front-facing camera that rotated to film each angle in turn. The company said the change came from feedback from its creative partners, because it let a director work behind the camera in the traditional way and gave tighter control over the filmed volume. UploadVR described the camera used to shoot Hallelujah in 2017 as a hexagonal rig with 95 lenses, which recorded the scene five times from five angles.

Immerge 2.0 (2017)

Lytro revealed Immerge 2.0 to Road to VR in November 2017. It kept roughly the same form, but its alternating rows of cameras pointed off-axis in opposite directions, so the lenses worked together as one wider camera. DPReview reported that the rig had 95 cameras in total.

Feature	Immerge (planar, 2017)	Immerge 2.0 (November 2017)
Field of view per capture	90 degrees	120 degrees
Rotations for a 360-degree scene	Five	As few as three
Calibration	Not stated	Automatic calibration process
Hallelujah output resolution	3.5K per eye	5K per eye (remaster)

The wider view also gave actors more room to perform. Lytro said it was ready to raise output to 10K per eye once headsets had the pixel density to show it, and that many of its software improvements could be applied to light field data captured before the changes. No Film School wrote that the change could let a 360-degree shoot use three units instead of four or five.

Software and features

Lytro built Immerge as a complete pipeline: camera, server, light field editor and player. The aim was to let creators combine computer graphics and live-action footage in a single piece. Lytro planned light field plugins for standard visual effects and editing tools, namely Nuke, Final Cut Pro, Adobe Premiere and Avid Media Composer. Road to VR reported that footage could be brought into Nuke and CGI elements placed within the scene, while plugins for Final Cut and Premiere were still in development at announcement.

The finished pieces were viewed on common virtual reality headsets through Lytro's own light field video playback engine. Engadget named the Oculus Rift and HTC Vive as supported headsets, and UploadVR also listed PlayStation VR. Road to VR reported that the player would stream compressed, edited light field footage from the cloud. It used predictive buffering, rendering only the area in view plus extra data for quick head movements, with a lower-detail spherical view as a fallback for large unexpected movements. Rosenthal said the viewer could scale down to low-power hardware such as the Samsung Gear VR.

File size was a practical problem. During the August 2016 demo, a 30-second clip took 50 GB and needed a powerful computer to play at 90 frames per second; Lytro told UploadVR it was aiming for 5 GB for the same clip by the end of that year. Lytro's VP of Engineering Tim Milliron told Road to VR that the system could render up to 8K per eye by synthesizing views from hundreds of sub-cameras, and that Lytro expected creators to treat captures as a master file for 6DoF desktop VR apps or ordinary 360-degree video. In October 2017 Lytro also announced Volume Tracer, software that generated light fields from pre-rendered CG content.

Productions

Moon (2016)

Moon was the demo Lytro showed in August 2016 as its first Immerge footage. UploadVR's reviewer could lean in around the scene, which he saw as the main difference from standard 360-degree video. Milliron explained that the CG elements were also rendered as light fields, so their reflections changed with the viewer's position.

Hallelujah (2017)

Hallelujah is a five-minute VR piece by Zach Richter, Bobby Halvorson and Eames Kolar, produced by Within with Lytro. It had its world premiere in the Virtual Arcade at the 2017 Tribeca Film Festival, where it was presented in six degrees of freedom "using Lytro Immerge technology". The piece is an a cappella arrangement of Leonard Cohen's song performed by Halvorson, with the International Orange Chorale credited as the choir, and its setting is St. Ignatius church in San Francisco. UploadVR reported recording at about 10K per eye and processing down to about 6K per eye for delivery. It added that viewers could move with 6DoF, though in a smaller volume than engine-built experiences made in Unity or Unreal Engine. Road to VR and DPReview later gave the original release's output as 3.5K per eye. Rosenthal called it "the first example of what things might be able to look like when you're doing true volumetric capture."

Lytro presented the piece at SIGGRAPH 2017 as "Hallelujah: The World's First Lytro VR Experience". The abstract described Immerge as the first live-action capture system to enable 6DoF playback, with parallax, view-dependent lighting and stereo in all directions. On 21 September 2017 Hallelujah was released to the public in the Within app for headsets including the Rift, Vive and Gear VR. That version was pre-rendered and did not support full movement; viewers could only turn their heads. UploadVR also reported that the full experience would be shown in an exhibit dedicated to Cohen at the MAC museum in Montreal from fall 2017.

Reception

Early hands-on coverage was positive about the parallax effect but noted image problems. In UploadVR's August 2016 hands-on of Moon, Azad Balabanian wrote that the effect "felt quite natural", with strong depth and parallax, but that he "noticed visible artifacts around the live-action cast, resulting in less than crisp outlines of the figures in the scene." He found a demo about six months newer much improved, and concluded that "lightfield video is still in its earliest stages".

After seeing the remastered Hallelujah side by side with the original, Road to VR's Ben Lang noted "a much cleaner outline around the main character", sharper textures and better stereoscopy on distant details. He wrote that the Immerge camera, though "a beast of a rig", was "capable of capturing some of the best looking volumetric video that I've had my eyes on yet." Road to VR's Scott Hayden later wrote that Immerge 2.0, despite its wider field of view and more capture elements, was still very large and was not close to the 360-degree rig form factor Lytro had first envisioned.

Discontinuation

On 20 March 2018 TechCrunch, citing unnamed sources, reported that Google was acquiring Lytro in what one source called an "asset sale" for no more than 40 million US dollars, while another source put the price at 25 million US dollars. Neither company commented, and the figures were never confirmed. On 27 March 2018 Lytro said it would stop "taking on new productions or providing professional services" while it wound down the company. TechCrunch reported that Google was not acquiring the company, and Google confirmed to TechCrunch that some Lytro team members were joining it. A source quoted by The Verge, as summarized by DPReview, described a "hiring deal" in which "a large fraction" of Lytro's employees would join Google, with no specific plans to use the acquired assets in current light field projects. Road to VR reported that the staff would be spread across several divisions, that Lytro's camera work would not continue, and that the company's website had been reduced to a support page.

See also

• Lytro
• Light field camera
• Volumetric video
• Cinematic VR
• Six degrees of freedom