Toggle search
Toggle menu
5K
5.9K
229
47.7K
VR & AR Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

VR photography

From VR & AR Wiki
More actions

VR photography is the capture and interactive viewing of panoramic still photographs that surround the viewer, usually covering a full 360-degree circle or a complete sphere, so that a person can look around the photographed scene on a screen or inside a head-mounted display. Such images are also called photo spheres, 360-degree images, or panoramic, spherical or omnidirectional images. They are made either by stitching together several overlapping photographs taken from one point, or in a single exposure with a multi-lens camera. A complete spherical image covers 360 degrees horizontally and 180 degrees vertically.[1][2]

The technique began as an image-based alternative to modeled 3D graphics. Apple's QuickTime VR, described by Shenchang Eric Chen at SIGGRAPH 1995, was designed to play back warped cylindrical panoramas and rotatable "object movies" at interactive speed on most personal computers without hardware acceleration.[3] Panoramic stills were among the first experiences Oculus announced for the Samsung Gear VR in 2014, and the field has since expanded to stereoscopic panoramas, 180-degree 3D stills and Apple's spatial photos. The term covers still images; moving footage is treated separately as 360 Video.

Reviewed 6 October 2026. Checked all paper citations (texts, abstracts, Crossref metadata) and the Google, Apple, Ricoh, NASA, IVRPA, Meta and press sources for dates, names, quotes and figures. About review dates.

Definition and scope

A VR photograph records the light arriving at a single viewpoint from many directions. When the camera only rotates about its optical center, which Chen calls the nodal point, all the frames can be mapped to one orientation-independent projection, an environment map, and any view from that point can be computed by reprojecting it.[3] The viewer can turn and zoom but cannot move through the scene, because a single panorama holds no information about how nearby objects would shift with a change of position.[3] Xu, Li, Zhang and Le Callet's 2020 survey describes 360-degree images (also called panoramic, spherical or omnidirectional images) as content on a sphere covering the whole 360 by 180 degree viewing range. A viewer wearing a head-mounted display picks which part of that sphere fills the viewport by moving the head.[2]

A stereoscopic panorama, unlike an ordinary monoscopic one, has slightly different views for each eye, so that near things look near and far things look far.[4] Panoramas need not cover the whole sphere: QuickTime VR's cylindrical panoramas had less than 180 degrees of vertical field of view, so the player did not let the user look straight up or down.[3] The VR180 format records a 180-degree field of view in 3D.[5]

Apple's spatial photos are a related but distinct format. Apple's developer documentation defines a spatial photo as a multi-image HEIC file containing a left-eye and a right-eye image plus metadata giving each camera's horizontal field of view, the baseline between the cameras and the projection. It also states that spatial media is always rectilinear, typically with a field of view below 90 degrees, and that equirectangular or fisheye content should not be encoded as spatial media.[6] Spatial photos are therefore stereo windows on a scene, not surround panoramas.

How it works

Capture methods

Method How it works Examples
Rotating a single camera Overlapping frames are shot while the camera turns about its optical center, then stitched. Chen's QuickTime VR workflow used a tripod-mounted camera centered at its nodal point, panned about the vertical axis in roughly equal steps and usually mounted sideways for maximum vertical coverage.[3] QuickTime VR authoring tools
Handheld mosaicing Szeliski and Shum's 1997 method stitched images from a hand-held camera without controlled motion, provided there was no strong motion parallax, and recovered 3D camera rotations directly.[7] Android Photo Sphere[8]
Fisheye shots A lens with a very large field of view, such as a fisheye lens, covers much of the scene in one frame.[7] Ford Oxaal's technology combined two or more fisheye or rectilinear images into one navigable spherical image, and NASA describes the IPIX package as able to capture an entire spherical environment with two shots.[9] IPIX
One-shot cameras A camera with a twin-lens optical system records the scene around, above and below the camera in a single shot.[10] Ricoh Theta (2013)[10]
Stereo sweep A phone is moved in a circle and the app builds a panorama with slightly different views for each eye. Cardboard Camera (2015)[4]
Stereo pairs Two cameras separated by a baseline record a left-eye and a right-eye image.[6] VR180 limits the view to 180 degrees in 3D. VR180 cameras such as the Lenovo Mirage Camera; iPhone spatial photos[5][11]

Stitching

Stitching aligns overlapping photographs and blends them into one seamless image. Brown and Lowe's 2007 paper Automatic Panoramic Image Stitching using Invariant Features describes a fully automatic method. It extracts SIFT features from every image, finds matching images with RANSAC and a probabilistic verification model, and then jointly refines every camera's rotation and focal length with bundle adjustment. The final steps are automatic straightening, gain compensation for exposure differences and multi-band blending. Because matching uses invariant features, the authors report that the method is insensitive to the order, orientation, scale and illumination of the input images, and that it can pick out several panoramas from an unordered set of photos.[12] The authors released a C++ implementation called Autostitch.[12]

The geometry assumes rotation about the optical center. Brown and Lowe note that panoramas "often suffer from parallax errors due to small motions of the optical centre", and they list parallax, mis-registration, radial distortion and vignetting among the causes of visible seams that blending has to hide.[12]

Projections and metadata

A spherical panorama has to be flattened to be stored as an ordinary image file. QuickTime VR stored cylindrical panoramas.[3] Google's photo sphere format uses the equirectangular projection; its XMP metadata (namespace http://ns.google.com/photos/1.0/panorama/, usually called GPano) records the projection type, the full panorama size and the position of any cropped region, and Google products support only the equirectangular value.[1] Szeliski and Shum argued that special cylindrical or spherical representations were unnecessary. Their system mapped the mosaic onto any texture-mapped polyhedron around the viewer, which avoided the singularities at the top and bottom of cylindrical and spherical maps and let standard 3D graphics hardware display the scene.[7]

The choice of projection has measurable side effects. In the survey by Xu and colleagues, the equirectangular projection (ERP) stretches content near the top and bottom borders of the image, while cubemap projection (CMP) has less geometric distortion than ERP. Because sphere-to-plane projections sample the sphere non-uniformly, ordinary metrics such as PSNR over-weight distortion where the sampling density is high; weighted-to-spherically-uniform PSNR (WS-PSNR) applies per-pixel weight maps to balance this.[2]

Stereoscopic panoramas

In Omnistereo: Panoramic Stereo Imaging (2001), Shmuel Peleg, Moshe Ben-Ezra and Yael Pritch defined an omnistereo panorama as a pair of panoramas, one per eye, that gives a stereo sensation over a full 360 degrees. Such panoramas can be built by mosaicing images from a single rotating camera, which also lets the stereo baseline be varied with scene distance, but a rotating camera limits capture to stationary scenes. The paper proposed a mirror and a lens that produce the same rays without moving parts.[13]

Consumer stereo stills followed this rotating-camera idea or avoided the problem by narrowing the field of view. Google's Cardboard Camera built three-dimensional panoramas "with slightly different views for each eye" from a phone swept in a circle.[4] In its coverage, Road to VR wrote that the result "gives some parallax, but it's not what you'd call perfect 3D" given that it comes from a single phone lens, and that the app blurs the top and bottom of the image.[14] VR180 stills and spatial photos avoid the full-sphere problem by covering only part of the scene: VR180 records a 180-degree field of view in 3D, and spatial photos are rectilinear stereo pairs.[5][6] Apple notes that a camera baseline near the average human eye separation of 64 mm produces content that feels real-world size in immersive presentation, while the cameras that record spatial video on iPhone 15 Pro are 19.2 mm apart.[6]

History

QuickTime VR was an early commercial VR photography system. Chen's 1995 paper presented it as a way around the laborious modeling and special rendering hardware of 3D virtual environments: a cylindrical panorama is warped on the fly to simulate panning and zooming, users "hop" between panoramic nodes, and orientation-independent hot spots link nodes or trigger events. The panoramas could be rendered, shot with panoramic cameras or stitched from overlapping photographs.[3] In 1997 Szeliski and Shum noted that most commercial products of this kind, including QuickTime VR and Surround Video, used cylindrical images with limited vertical coverage, while newer systems supported full spherical maps.[7]

Spherical capture from a small number of fisheye images was commercialized by a company in Knoxville, Tennessee, that began as Telerobotics International. It changed its name to Omniview in 1995 after Ford Oxaal showed his spherical media technology, which combined two or more fisheye or rectilinear images into one navigable spherical image. It became Interactive Pictures Corporation in 1998, then Internet Pictures Corporation, and finally IPIX Corporation. According to NASA, a Small Business Innovation Research contract through Langley Research Center helped Interactive Pictures Corporation create the imaging technology.[9]

Practitioners also formed a professional association. The International QuickTime VR Association (IQTVRA) was incorporated in California on 7 December 1998, changed its name to IVRPA in December 2004 to reflect a broadening field of immersive interactive imaging, and on 1 June 2018 became the International Virtual Reality Professionals Association to cover 360 video as well as still media.[15]

In the 2010s, phone apps and one-shot cameras made spherical capture available to consumers. Android 4.2, announced on 29 October 2012, added Photo Sphere, which Google described as letting users "snap shots up, down and in every direction to create stunning 360-degree immersive experiences" for sharing on Google+ or adding to Google Maps.[8][16] On 5 September 2013 Ricoh announced the RICOH THETA, which it called "the world's first mass-produced imaging device that encapsulates fully spherical scenes with one shot". It used a twin-lens folded optical system, weighed about 95 g and was priced from US$399 (pre-tax) in the United States, with pre-orders in September and general sales through Ricoh's online channels from October 2013.[10]

Headsets then gave these images a native display. Oculus announced Oculus 360 Photos, an app for playing back panoramic content in VR, with the Samsung Gear VR Innovator Edition on 3 September 2014; see Oculus 360 Photos.[17] Google released Cardboard Camera on 3 December 2015.[4] Facebook launched 360 Photos on the web, Android and iOS on 9 June 2016. It turned panoramas and 360-camera images into photos that could be explored by tilting a phone, swiping or dragging, with a "View in VR" option for Gear VR owners; organizations sharing 360 Photos at launch included the New York Times and NASA.[18][19]

Stereo stills became more common later. In May 2018 Google described VR180 photos and videos as having "a 180 degree field of view and crisp, three-dimensional imagery", named the Lenovo Mirage Camera as the first VR180 consumer camera, and added VR180 support to Google Photos.[5] Apple's iPhone 16 and iPhone 16 Plus, announced on 9 September 2024, capture spatial photos and videos for viewing on Apple Vision Pro.[11]

Viewing in VR and AR

In a head-mounted display, the viewer's head movement chooses which part of the sphere fills the viewport.[2] Because the image is tied to one viewpoint, it can be reprojected for any viewing direction but not for a change of position, so the scene follows head rotation but not head translation.[3] Facebook's 360 Photos let the same image be explored in the News Feed on a phone or, through a "View in VR" option, inside Gear VR.[18] On Apple Vision Pro, spatial photos appear by default in a window with the stereo content inset, and the viewer can choose an immersive mode that shows the scene at real-world scale. Apple says both modes include visual treatments meant to reduce common causes of stereo viewing discomfort.[6]

Viewing behavior in VR differs from desktop viewing. Sitzmann and colleagues recorded 1,980 head and gaze trajectories from 169 people exploring 22 static stereoscopic omnidirectional panoramas in a head-mounted display. Fixations clustered strongly around the equator of the panoramas, an "equator bias" that the authors used to adapt existing saliency predictors to VR.[20] The authors note that most of their scenes had a clear horizon line, which may have contributed to the effect.[20]

Applications

Virtual tours. Linked panoramas with hot spots were already part of QuickTime VR as Chen described it in 1995.[3] The IVRPA lists 360-degree panoramas, 360-degree virtual tours and gigapixel panoramas among the content its members produce.[15]

Object photography. QuickTime VR's object movies stored a two-dimensional array of frames shot from different directions so that a user could grab an object with the mouse and turn it. Chen contrasts the two players: the panoramic player looks around a space from the inside, while the object player views an object from the outside.[3]

Media and social sharing. Google presented Photo Sphere images as something to share on Google+ or add to Google Maps.[8] Facebook's 360 Photos launched with organizations such as the New York Times and NASA sharing content, and Google said VR180 support in Google Photos let people share VR180 captures with friends and family regardless of their device.[18][5]

Lighting for mixed reality. Omnidirectional photographs are also used to light computer graphics. Paul Debevec's SIGGRAPH 98 paper Rendering Synthetic Objects into Real Scenes used a light probe to measure the incident illumination where synthetic objects were to be placed, and composited the rendered objects into a photograph of the scene.[21] Debevec defines a light probe image as "an omnidirectional, high dynamic range image that records the incident illumination conditions at a particular point in space", and notes that some of his probes were assembled from high-dynamic-range panoramas.[22] AR frameworks apply the same idea in real time: Apple's ARKit can generate environment textures, which "depict the view in all directions from a specific point in a scene", from camera imagery during an AR session and use them for image-based lighting of virtual objects.[23]

Research

One line of research adds motion parallax, so that a photographed scene responds when the viewer moves the head as well as turns it. Hedman, Alsisan, Szeliski and Kopf's Casual 3D Photography (2017) reconstructs a "3D photo", a central panoramic, multi-layered textured mesh, from photos taken with a hand-held cell phone or DSLR camera. The result can be rendered with a standard rasterization pipeline to produce perspective views with motion parallax, and in VR it gives geometrically consistent views for both eyes.[24] Hedman and Kopf's Instant 3D Photography (2018) builds 3D panoramas from the color-and-depth pairs produced by dual-lens phone cameras, at about one input image per second and about a minute end to end for mid-sized panoramas, with results viewable with binocular and head-motion parallax in VR.[25]

Bertel, Yuan, Lindroos and Richardt's OmniPhotos (2020) captures a 360-degree panorama with motion parallax from a single sweep of a consumer 360-degree video camera, which takes less than 3 seconds on a rotating selfie stick or 10 seconds handheld. The authors describe this as the fastest capture time "for any VR photography approach supporting motion parallax by an order of magnitude", and they fit a deformable proxy geometry to a sparse 3D reconstruction to reduce vertical distortion.[26]

Display at very high resolution is a separate problem. Lyu and colleagues note that a gigapixel panorama cannot simply be loaded onto a GPU because of limited texture memory. Their out-of-core renderer uses hierarchical tiling and on-demand loading to show such panoramas on head-mounted displays at more than 50 frames per second, even on low-end GPUs, and it adds refocusing driven by an embedded gaze tracker.[27] Work on perception, quality assessment and compression of 360-degree images is reviewed in the survey by Xu, Li, Zhang and Le Callet.[2]

See also

References

  1. ↑ 1.0 1.1 "Photo Sphere XMP Metadata". Google for Developers. Google. https://developers.google.com/streetview/spherical-metadata. Retrieved 2026-10-06.
  2. ↑ 2.0 2.1 2.2 2.3 2.4 Mai Xu, Chen Li, Shanyi Zhang, Patrick Le Callet (2020). "State-of-the-Art in 360° Video/Image Processing: Perception, Assessment and Compression". IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 1, pp. 5-26. doi:10.1109/JSTSP.2020.2966864. https://doi.org/10.1109/JSTSP.2020.2966864. Retrieved 2026-10-06.
  3. ↑ 3.00 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09 Shenchang Eric Chen (1995). "QuickTime VR: An Image-Based Approach to Virtual Environment Navigation". SIGGRAPH '95: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 29-38. ACM. doi:10.1145/218380.218395. https://doi.org/10.1145/218380.218395. Retrieved 2026-10-06.
  4. ↑ 4.0 4.1 4.2 4.3 Carlos Hernandez (2015-12-03). "Step inside your photos with Cardboard Camera". The Keyword (Google blog). Google. https://blog.google/products/google-ar-vr/step-inside-your-photos-with-cardboard/. Retrieved 2026-10-06.
  5. ↑ 5.0 5.1 5.2 5.3 5.4 Clay Bavor (2018-05-04). "Introducing the first Daydream standalone VR headset and new ways to capture memories". The Keyword (Google blog). Google. https://blog.google/products-and-platforms/products/google-ar-vr/new-daydream-headset-and-vr180-camera-now-available/. Retrieved 2026-10-06.
  6. ↑ 6.0 6.1 6.2 6.3 6.4 "Creating spatial photos and videos with spatial metadata". Apple Developer Documentation. Apple. https://developer.apple.com/documentation/imageio/creating-spatial-photos-and-videos-with-spatial-metadata. Retrieved 2026-10-06.
  7. ↑ 7.0 7.1 7.2 7.3 Richard Szeliski, Heung-Yeung Shum (1997). "Creating Full View Panoramic Image Mosaics and Environment Maps". SIGGRAPH '97: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 251-258. ACM. doi:10.1145/258734.258861. https://doi.org/10.1145/258734.258861. Retrieved 2026-10-06.
  8. ↑ 8.0 8.1 8.2 Andy Rubin (2012-10-29). "Nexus: The best of Google, now in three sizes". The Keyword (Google blog). Google. https://blog.google/products/nexus/nexus-best-of-google-now-in-three-sizes/. Retrieved 2026-10-06.
  9. ↑ 9.0 9.1 "Immersive Photography Renders 360º Views". NASA Spinoff 2008. NASA. 2008. https://spinoff.nasa.gov/Spinoff2008/ch_5.html. Retrieved 2026-10-06.
  10. ↑ 10.0 10.1 10.2 "Ricoh to Market Mass-produced Imaging Device for Fully Spherical Imagery". RICOH THETA news archive. Ricoh Company, Ltd.. 2013-09-05. https://theta-archive.ricoh360.com/topics.theta360.com/news/2013-09-05/. Retrieved 2026-10-06.
  11. ↑ 11.0 11.1 "Apple introduces iPhone 16 and iPhone 16 Plus". Apple Newsroom. Apple. 2024-09-09. https://www.apple.com/newsroom/2024/09/apple-introduces-iphone-16-and-iphone-16-plus/. Retrieved 2026-10-06.
  12. ↑ 12.0 12.1 12.2 Matthew Brown, David G. Lowe (2007). "Automatic Panoramic Image Stitching using Invariant Features". International Journal of Computer Vision, vol. 74, no. 1, pp. 59-73. doi:10.1007/s11263-006-0002-3. https://doi.org/10.1007/s11263-006-0002-3. Retrieved 2026-10-06.
  13. ↑ Shmuel Peleg, Moshe Ben-Ezra, Yael Pritch (2001). "Omnistereo: Panoramic Stereo Imaging". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 279-290. doi:10.1109/34.910880. https://doi.org/10.1109/34.910880. Retrieved 2026-10-06.
  14. ↑ Scott Hayden (2015-12-03). "Google's 'Cardboard Camera' Lets You Snap 3D Photos from Your 2D Phone Camera". Road to VR. https://www.roadtovr.com/googles-cardboard-camera-lets-snap-3d-photos-2d-phone-camera/. Retrieved 2026-10-06.
  15. ↑ 15.0 15.1 "About IVRPA". IVRPA. https://ivrpa.org/about-ivrpa/. Retrieved 2026-10-06.
  16. ↑ Jordan Crook (2012-10-29). "Android 4.2 Jelly Bean Has Arrived: Photo Sphere Panoramic Camera, Gesture Typing, Wireless HDTV Streaming". TechCrunch. https://techcrunch.com/2012/10/29/android-4-2-jelly-bean/. Retrieved 2026-10-06.
  17. ↑ "Introducing the Samsung Gear VR Innovator Edition". Meta Blog. Meta. 2014-09-03. https://www.meta.com/blog/introducing-the-samsung-gear-vr-innovator-edition/. Retrieved 2026-10-06.
  18. ↑ 18.0 18.1 18.2 Paul Sawers (2016-06-09). "Facebook launches 360 Photos to transform any panorama shot into an immersive experience". VentureBeat. https://venturebeat.com/mobile/facebook-launches-360-photos-so-anyone-can-transform-panoramas-into-immersive-photos/. Retrieved 2026-10-06.
  19. ↑ Josh Constine (2016-05-11). "Facebook will turn panoramas into "360 Photos" for feed and Gear VR's 1M users". TechCrunch. https://techcrunch.com/2016/05/11/facebook-360-photos/. Retrieved 2026-10-06.
  20. ↑ 20.0 20.1 Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, Gordon Wetzstein (2018). "Saliency in VR: How Do People Explore Virtual Environments?". IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 4, pp. 1633-1642. doi:10.1109/TVCG.2018.2793599. https://doi.org/10.1109/TVCG.2018.2793599. Retrieved 2026-10-06.
  21. ↑ Paul Debevec (1998). "Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography". SIGGRAPH '98: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, pp. 189-198. ACM. doi:10.1145/280814.280864. https://doi.org/10.1145/280814.280864. Retrieved 2026-10-06.
  22. ↑ Paul Debevec. "Light Probe Image Gallery". pauldebevec.com. https://www.pauldebevec.com/Probes/. Retrieved 2026-10-06.
  23. ↑ "AREnvironmentProbeAnchor". Apple Developer Documentation. Apple. https://developer.apple.com/documentation/arkit/arenvironmentprobeanchor. Retrieved 2026-10-06.
  24. ↑ Peter Hedman, Suhib Alsisan, Richard Szeliski, Johannes Kopf (2017). "Casual 3D Photography". ACM Transactions on Graphics, vol. 36, no. 6. doi:10.1145/3130800.3130828. https://doi.org/10.1145/3130800.3130828. Retrieved 2026-10-06.
  25. ↑ Peter Hedman, Johannes Kopf (2018). "Instant 3D Photography". ACM Transactions on Graphics, vol. 37, no. 4. doi:10.1145/3197517.3201384. https://doi.org/10.1145/3197517.3201384. Retrieved 2026-10-06.
  26. ↑ Tobias Bertel, Mingze Yuan, Reuben Lindroos, Christian Richardt (2020). "OmniPhotos: Casual 360° VR Photography". ACM Transactions on Graphics, vol. 39, no. 6. doi:10.1145/3414685.3417770. https://doi.org/10.1145/3414685.3417770. Retrieved 2026-10-06.
  27. ↑ Wentao Lyu, Peng Ding, Yingliang Zhang, Anpei Chen, Minye Wu, Shu Yin, Jingyi Yu (2021). "Refocusable Gigapixel Panoramas for Immersive VR Experiences". IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 3, pp. 2028-2040. doi:10.1109/TVCG.2019.2940444. https://doi.org/10.1109/TVCG.2019.2940444. Retrieved 2026-10-06.
Retrieved from "https://vrarwiki.com/index.php?title=VR_photography&oldid=47430"