Toggle search
Toggle menu
4.9K
5.5K
229
46.8K
VR & AR Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Projection mapping

From VR & AR Wiki
More actions

Projection mapping (PM) is the use of projectors to overlay computer-generated imagery onto physical surfaces, including non-planar and textured objects, so that the projected light is registered to the shape of the surface. Daisuke Iwai, in a 2024 review of the field, describes it as creating an augmented reality environment "where the virtual and real worlds seamlessly merge", and notes that projection-based AR is often called spatial augmented reality (SAR).[1] The same technique is also called projected augmented reality or video mapping.[2]

Iwai describes projection mapping as one of the fundamental approaches to building AR systems. Unlike video see-through and optical see-through displays, it does not require users to wear a head-mounted display or hold a phone, it does not restrict their field of view, and several people can see the same augmentation at once.[1][3] Early research on the technique includes the Office of the Future project at the University of North Carolina at Chapel Hill and the 1998 paper there that named "Spatially Augmented Reality".[4][5]

Reviewed 4 October 2026. Checked every claim against the Iwai 2016 and 2024 papers, the 1998 SAR and Office of the Future papers, Shader Lamps, iLamps, ShareVR, Makeup Lamps and Lightform PDFs, Crossref DOI metadata, the PubMed MIPS abstract, arXiv records, Microsoft Research pages, naimark.net, Dezeen and lightform.com. About review dates.

Terminology

Several names are in use. Ramesh Raskar, Greg Welch and Henry Fuchs introduced "Spatially Augmented Reality (SAR)" in 1998 as "a new paradigm" in which "virtual objects are rendered directly within or on the user's physical space", for example by using digital light projectors to "paint" imagery onto real surfaces or by using built-in flat panel displays.[4] Oliver Bimber and Raskar used the form "spatial augmented reality" as the title of their 2005 book on projector-based and other spatial AR displays.[6] Authors from Lightform, a projection mapping hardware company, wrote in 2018 that "projected augmented reality, also called projection mapping or video mapping" uses projected light to augment 3D surfaces directly instead of pass-through screens or headsets.[2] Raskar's 1998 definition of SAR also covered flat panel displays embedded in the environment, so SAR is a broader term than projection mapping, which uses projectors.[4]

How it works

A projection mapping system has to solve the problems that arise when an ordinary projector is pointed at something other than a flat white screen. Iwai's 2024 review groups the technical work into four areas: geometric registration, radiometric compensation, defocus compensation and shadow removal.[1]

Geometric registration

The system must know which projector pixel lands on which surface point. Conventional keystone correction handles only flat surfaces, so projection mapping describes the projector with the pinhole model used for cameras: a surface point (X, Y, Z) maps to a projector pixel (x, y) through a 3 x 4 matrix of intrinsic parameters (such as focal length) and a 4 x 4 matrix of extrinsic parameters (the projector's pose relative to the surface). For static setups these parameters are usually estimated by having a camera capture projected calibration patterns, building on standard camera calibration methods.[1] Conventional multi-projector calibration projects and captures structured light patterns for each projector in turn, so calibration time grows with the number of projectors.[7] The 1998 Office of the Future paper proposed projectors that would capture the geometry of office surfaces with imperceptible structured light while also displaying images on those surfaces.[5]

The 2001 Shader Lamps prototype is an early example. Each of two Sony VPL6000U projectors (1024 x 768) was calibrated by moving a projected cross-hair onto about 20 known 3D fiducials on the physical model; a 3 x 4 perspective projection matrix was then computed and decomposed into intrinsic and extrinsic parameters. The authors reported that calibration took under five minutes per projector and that re-projection error was typically less than two pixels; they contrasted this with the alignment process of earlier systems, which could take several hours even for a single projector.[8] Later systems automated the step. Lightform's LF1, a commercial device, attached a camera to the projector and ran a visible structured light scan to obtain a dense projector-to-camera pixel correspondence.[2]

Radiometric compensation

Even when every pixel lands in the right place, a colored or textured surface changes the color the viewer sees. Radiometric (photometric) compensation computes the projector input needed to produce a target color at each surface point, by modeling how a projector value becomes an observed color and inverting that model. Early work used linear color transformation models; later studies used nonlinear models such as thin-plate splines, and since about 2019 deep neural networks have been trained to perform the compensation, in some cases together with geometric registration.[1] When several projectors overlap, their intensities must also be blended. Shader Lamps introduced a feathering method that assigns per-pixel weights while ignoring paths that cross depth discontinuities, so that overlaps stay smooth even when the projectors' color responses differ.[8]

Defocus, shadows and hardware limits

Projectors use large apertures to maximize brightness, so they have a shallow depth of field and parts of a deep object appear blurred. Users and objects between the projector and the surface also cast shadows, which researchers address by illuminating the occluded area from additional projectors or with optical methods.[1] The Shader Lamps authors listed the main limitations of the approach as its dependence on a neutral physical surface and dark ambient lighting, secondary light scattering, the limited depth of field and black level of projectors, and shadows cast by users.[8]

Dynamic and view-dependent projection

In dynamic projection mapping the target object moves, so its pose must be tracked and the image re-rendered fast enough that misalignment is not visible. Iwai cites a perceptual threshold of about 6-7 ms from motion to projection, which is why typical 60 Hz projectors are poorly suited to the task; high-speed projectors capable of almost 1,000 frames per second of full-color video, paired with cameras of similar speed, are used in research systems.[1]

Projecting onto a surface changes the surface's appearance, but by itself cannot make an object appear to float in front of it. With head tracking and head-coupled perspective rendering, the image can be made correct for one viewer's eye position. The 1998 SAR paper analyzed how registration errors behave in this case: as in a CAVE, viewpoint orientation error has almost no effect, while viewpoint position error produces a kind of image shear rather than a simple misregistration. It also noted that a real object such as the user's hand can occlude a projected virtual object, but a virtual object cannot hide a real one.[4] Stereoscopic projection mapping adds time-sequential left and right images viewed through active-shutter glasses so that 3D objects can appear above arbitrary surfaces. Because the eyes still focus on the physical surface, this causes a vergence-accommodation conflict, which researchers have tried to reduce with multifocal designs that use electrically focus-tunable lenses.[1]

History

The Shader Lamps paper cites several precedents from theater and theme parks: the projection of pre-recorded video onto four neutral busts of singing men in the Walt Disney World "Haunted Mansion", a patented projector and fiber-optic setup that animates the head of the fortune teller "Madame Leota" inside a crystal ball, and the Son et Lumiere light show on the Blois castle in France, which projects slides onto architecture.[8]

Michael Naimark produced the immersive film installation Displacements three times between 1980 and 1984. A living room was filmed with two performers using a 16 mm camera on a slowly rotating turntable; the camera was then replaced by a film loop projector and the whole room was spray-painted white, so the footage was projected back onto the objects it showed. Naimark wrote that everything appeared "strikingly 3D" except the people, who looked ghostlike. The last of the original versions was shown at the San Francisco Museum of Modern Art in 1984, and a digital video version followed in 2005.[9][8]

At UNC Chapel Hill, the Office of the Future paper, presented at SIGGRAPH 98, proposed replacing ceiling lights with computer-controlled cameras and "smart" projectors that would scan the walls, furniture and people of an office and turn everyday, possibly irregular, surfaces into displays. It described the system as building on the CAVE, tiled displays and image-based modeling, and it used a two-pass projective texture method to render images that appear correct to a moving, head-tracked viewer.[5] The CAVE itself, developed by Carolina Cruz-Neira, Daniel J. Sandin and Thomas A. DeFanti, is a surround-screen, projection-based virtual reality system that projected onto dedicated screens rather than onto everyday objects.[10] Raskar, Welch and Fuchs wrote that the Office of the Future work led them to the SAR paradigm, after they realized that using irregular surfaces as displays meant registering 2D imagery with 3D physical geometry.[4]

Shader Lamps (Raskar, Welch, Kok-Lim Low and Deepak Bandyopadhyay, 2001) proposed replacing a physical object's color, texture and material with "a neutral object and projected imagery", so that the visual properties of the object are "lifted" into the projector. Its demonstrations included a physical model of the Taj Mahal (21,000 triangles and 15 texture maps) and a clay vase that, with a tracked viewer, could be made to look like metal or plastic.[8] At Mitsubishi Electric Research Laboratories, Raskar and colleagues then presented iLamps at SIGGRAPH 2003: projectors with cameras that could determine the geometry of the display surface, augment objects from a hand-held unit, and form self-configuring clusters on non-planar surfaces.[11]

Bimber and Raskar's book Spatial Augmented Reality: Merging Real and Virtual Worlds (A K Peters, 2005) collected rendering algorithms, calibration methods and display examples for AR that uses large optical elements and video projectors instead of worn or hand-held displays.[6] Comprehensive surveys of projector-camera systems and projection mapping followed in Computer Graphics Forum in 2008 and 2018, the latter a state-of-the-art report by Anselm Grundhöfer and Iwai covering self-calibration, radiometric compensation and computational displays that work around projector limits in dynamic range, refresh rate, resolution, depth of field and color space.[1][12][13]

Selected milestones

Year Work Authors or organization Contribution
1980-1984 Displacements Michael Naimark Film of a room projected back onto the same room painted white[9]
1998 The Office of the Future UNC Chapel Hill Cameras and projectors to scan and display on everyday office surfaces[5]
1998 Spatially Augmented Reality Raskar, Welch, Fuchs Named and analyzed projector-based AR with head tracking[4]
2001 Shader Lamps Raskar, Welch, Low, Bandyopadhyay Projected materials on neutral objects; multi-projector blending[8]
2003 iLamps Mitsubishi Electric Research Laboratories Geometry-aware, hand-held and clustered projectors[11]
2005 Spatial Augmented Reality (book) Bimber, Raskar Reference text on spatial AR displays[6]
2013 IllumiRoom Microsoft Research Kinect and projector extend TV games into the room[14]
2014 RoomAlive Microsoft Research Networked projector-depth camera units map interactive content onto a whole room[15]
2017 Makeup Lamps Disney Research and partners Marker-less projection onto live performers' faces[16]
2018 Lightform LF1 Lightform, Inc. Projector-mounted scanner and authoring software for projected AR[2]

Comparison with head-worn AR

Aspect Projection mapping (SAR)
Display worn or held by the user None; head-mounted displays or smartphones are not needed[1]
Field of view Not restricted by a display[1]
Shared viewing Several people can share the same in-situ augmentation at once[1]
Content floating above surfaces Not possible with projection alone; needs head tracking and stereoscopic glasses[1]
Occlusion Real objects can hide virtual ones, but virtual objects cannot hide real ones[4]
Tracking error Viewpoint orientation error has little effect; position error appears as image shear[4]
Environment Depends on neutral surfaces and dark ambient light; users can cast shadows on the projection[8]

Applications in VR and AR

Room-scale and living-room experiences

Microsoft Research's IllumiRoom (2013) combined a projector and a Kinect for Windows camera to augment the area surrounding a television screen. Using the room's geometry and appearance captured by the Kinect, it adapted projected visuals in real time to extend the field of view, change the appearance of the room, induce apparent motion and enable new game experiences.[14] The paper by Brett Jones, Hrvoje Benko, Eyal Ofek and Andy Wilson received a Best Paper award at CHI 2013.[17] RoomAlive (UIST 2014) extended the idea to a whole room with auto-calibrating, self-localizing projector-depth camera units that built a unified model of the room, so users could touch, shoot, stomp, dodge and steer projected content.[15] Microsoft released calibration code and Unity scripts from its projection mapping projects as the open-source RoomAlive Toolkit, which calibrates multiple projectors and Kinect cameras; its example RoomAlive scene used six projectors and six Kinect cameras.[18]

Sharing VR with bystanders

Research systems have used projection to include people who are not wearing a VR headset. ShareVR (CHI 2017), from Ulm University, combined an HTC Vive with two short-throw projectors that showed the virtual environment on the floor of the tracking space, plus a tracked hand-held display that served as a "window into the virtual world" for non-HMD users. In a study with 16 participants, the authors reported higher enjoyment, presence and social interaction than in a baseline condition in which the non-HMD user played with a gamepad and a television.[19] Later research mounted projectors on the headset itself: HMD Light (UIST 2020) projected the scene a VR user was watching onto the floor around them, and AAR (UIST 2020) added an actuated head-mounted projector to an optical see-through AR headset to share its content with nearby people and show extra interface elements.[1]

Hybrid systems with head-mounted displays

Projection and head-worn displays can complement each other. Projection alone cannot reproduce view-dependent effects such as specular reflections for several observers at once, so HySAR (2018) displayed those view-dependent components on an optical see-through display worn by each observer, in combination with spatial AR projection; the authors also showed that the combination extends the contrast of the result.[1] Raskar, Welch and Fuchs had already speculated in 1998 about combining SAR with see-through AR.[4]

Entertainment, performance and design

Disney Research's Makeup Lamps (Eurographics 2017) altered the appearance of live performers' faces. The system tracked facial pose and expression with infrared illumination and an aligned high-speed camera, predicted motion with adaptive Kalman filtering, and reached an average system latency of 9.8 ms; the authors described it as the first method for dynamic facial projection mapping without physical tracking markers that also handles facial expressions.[16] Earlier, the Shader Lamps authors proposed uses in architecture and city planning (virtual shadows on scaled models), engineering markings such as drilling locations, stage shows, and checking sculptors' clay models.[8] Lightform's LF1, a camera and computer that attached to a projector and was paired with the Lightform Creator software, was named Digital Design of the Year at the 2018 Dezeen Awards.[2][20] Lightform's website now lists the LF1 as discontinued.[21]

Medicine

Projection mapping has been tested for surgical guidance; Iwai's review cites it as a way to make an invisible signal marking the resection area visible on the organ.[1] At Kyoto University, surgeons and co-authors from Panasonic AVC Networks Company developed the Medical Imaging Projection System (MIPS), which uses the indocyanine green emission signal and active projection mapping for liver resection.[22] In a retrospective comparison of 23 patients who underwent anatomic hepatectomy with MIPS and 29 without it between September 2014 and September 2015, there were no significant differences in surgical or clinical outcomes; the demarcation lines were clearly projected in 21 of the 23 patients and were undetectable in 2. The authors concluded that their analysis provided evidence of the feasibility and clinical utility of MIPS for identifying anatomical landmarks during parenchymal dissection.[22]

Research

Iwai's 2024 review describes a shift from static to dynamic projection mapping, the use of invisible infrared markers or embedded infrared LEDs for tracking, deep-network radiometric compensation, wearable and omnidirectional projectors, and perceptual tricks that make physically impossible appearances look correct to observers.[1] Neural projection mapping (Yotam Erel, Iwai and Amit Bermano, 2023) places a virtual projector inside a neural reflectance field so that the projector can be calibrated and the projected texture optimized for a desired appearance from new viewpoints.[23] A 2026 preprint by Takumi Kawano, Kohei Miura and Iwai embeds cameras in the calibration target so that patterns from many projectors can be captured at once, reducing the number of projection-capture cycles from linear to nearly constant in the number of projectors.[7] Another 2026 preprint, FF-ProCams, uses a feed-forward Gaussian splatting model to predict a relightable scene representation for projector-camera systems from 8 input views, where the optimization-based baselines it was compared with used 297; its authors describe such systems as core infrastructure for spatial AR and projection mapping.[24]

See also

References

  1. ↑ 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 1.12 1.13 1.14 1.15 1.16 Daisuke Iwai (2024-03-11). "Projection mapping technologies: A review of current trends and future directions". Proceedings of the Japan Academy, Series B, vol. 100, no. 3, pp. 234-251. https://doi.org/10.2183/pjab.100.012. Retrieved 2026-10-04.
  2. ↑ 2.0 2.1 2.2 2.3 2.4 Brittany Factura, Laura LaPerche, Phil Reyneri, Brett Jones, Kevin Karsch (2018). "Lightform: Procedural Effects for Projected AR". SIGGRAPH '18 Studio (arXiv:2001.00521). doi:10.1145/3214822.3214823. https://arxiv.org/pdf/2001.00521. Retrieved 2026-10-04.
  3. ↑ Daisuke Iwai (2016). "Projection Mapping Technologies for AR". Proceedings of International Display Workshops (IDW '16), pp. 1076-1078. https://arxiv.org/abs/1704.02897. Retrieved 2026-10-04.
  4. ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 Ramesh Raskar, Greg Welch, Henry Fuchs (1998-11-01). "Spatially Augmented Reality". First International Workshop on Augmented Reality (IWAR '98), San Francisco. https://web.media.mit.edu/~raskar/UNC/Office/0~IWAR_SAR.pdf. Retrieved 2026-10-04.
  5. ↑ 5.0 5.1 5.2 5.3 Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, Henry Fuchs (1998). "The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays". Proceedings of SIGGRAPH 98, pp. 179-188. ACM. doi:10.1145/280814.280861. https://web.media.mit.edu/~raskar/UNC/Office/future_office.pdf. Retrieved 2026-10-04.
  6. ↑ 6.0 6.1 6.2 Oliver Bimber, Ramesh Raskar (2005). "Spatial augmented reality: merging real and virtual worlds". SearchWorks catalog, Stanford Libraries. A K Peters. https://searchworks.stanford.edu/view/6311161. Retrieved 2026-10-04.
  7. ↑ 7.0 7.1 Takumi Kawano, Kohei Miura, Daisuke Iwai (2026-04-27). "Breaking the Scalability Limit of Multi-Projector Calibration with Embedded Cameras". arXiv. https://arxiv.org/abs/2604.24024. Retrieved 2026-10-04.
  8. ↑ 8.0 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 Ramesh Raskar, Greg Welch, Kok-Lim Low, Deepak Bandyopadhyay (2001-06). "Shader Lamps: Animating Real Objects With Image-Based Illumination". Rendering Techniques 2001 (Eurographics Workshop on Rendering, London), MERL TR2001-21. Mitsubishi Electric Research Laboratories. https://www.merl.com/publications/docs/TR2001-21.pdf. Retrieved 2026-10-04.
  9. ↑ 9.0 9.1 Michael Naimark. "Displacements". naimark.net. http://www.naimark.net/projects/displacements.html. Retrieved 2026-10-04.
  10. ↑ Carolina Cruz-Neira, Daniel J. Sandin, Thomas A. DeFanti (1993). "Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE". Proceedings of SIGGRAPH '93, pp. 135-142. ACM. doi:10.1145/166117.166134. https://www.evl.uic.edu/documents/siggraph93-cave-cruz-neira.pdf. Retrieved 2026-10-04.
  11. ↑ 11.0 11.1 Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, Clifton Forlines (2003-07). "iLamps: Geometrically Aware and Self-Configuring Projectors". ACM Transactions on Graphics, vol. 22, no. 3 (SIGGRAPH 2003), pp. 809-818. Mitsubishi Electric Research Laboratories. doi:10.1145/882262.882349. https://www.merl.com/publications/docs/TR2003-23.pdf. Retrieved 2026-10-04.
  12. ↑ Oliver Bimber, Daisuke Iwai, Gordon Wetzstein, Anselm Grundhöfer (2008). "The Visual Computing of Projector-Camera Systems". Computer Graphics Forum, vol. 27, no. 8, pp. 2219-2245. https://doi.org/10.1111/j.1467-8659.2008.01175.x. Retrieved 2026-10-04.
  13. ↑ Anselm Grundhöfer, Daisuke Iwai (2018-05). "Recent Advances in Projection Mapping Algorithms, Hardware and Applications". Computer Graphics Forum, vol. 37, no. 2, pp. 653-675. https://doi.org/10.1111/cgf.13387. Retrieved 2026-10-04.
  14. ↑ 14.0 14.1 "IllumiRoom: Peripheral Projected Illusions for Interactive Experiences". Microsoft Research. Microsoft. 2013. https://www.microsoft.com/en-us/research/project/illumiroom-peripheral-projected-illusions-for-interactive-experiences/. Retrieved 2026-10-04.
  15. ↑ 15.0 15.1 Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew D. Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, Lior Shapira (2014-10). "RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-Camera Units". Microsoft Research (UIST '14, pp. 637-644). doi:10.1145/2642918.2647383. https://www.microsoft.com/en-us/research/publication/roomalive-magical-experiences-enabled-by-scalable-adaptive-projector-camera-units/. Retrieved 2026-10-04.
  16. ↑ 16.0 16.1 Amit H. Bermano, Markus Billeter, Daisuke Iwai, Anselm Grundhöfer (2017). "Makeup Lamps: Live Augmentation of Human Faces via Projection". Computer Graphics Forum, vol. 36, no. 2 (Eurographics 2017), pp. 311-323. Disney Research. doi:10.1111/cgf.13128. https://studios.disneyresearch.com/wp-content/uploads/2019/03/Makeup-Lamps-Live-Augmentation-of-Human-Faces-via-Projection-1.pdf. Retrieved 2026-10-04.
  17. ↑ "CHI 2013: an Immersive Event". Microsoft Research Blog. Microsoft. 2013-04-29. https://www.microsoft.com/en-us/research/blog/chi-2013-immersive-event/. Retrieved 2026-10-04.
  18. ↑ "RoomAlive Toolkit". GitHub. Microsoft. https://github.com/microsoft/RoomAliveToolkit. Retrieved 2026-10-04.
  19. ↑ Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, Enrico Rukzio (2017-05). "ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users". Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 4021-4033. ACM. doi:10.1145/3025453.3025683. https://www.uni-ulm.de/fileadmin/website_uni_ulm/iui.inst.100/1-hci/hci-paper/2017/2017-shareVr_small.pdf. Retrieved 2026-10-04.
  20. ↑ "Lightform LF1 by Lightform". Dezeen Awards 2018. Dezeen. 2018. https://www.dezeen.com/awards/2018/winners/lightform-lf1-lightform/. Retrieved 2026-10-04.
  21. ↑ "Lightform LF1 - Discontinued". Lightform. https://lightform.com/lf1. Retrieved 2026-10-04.
  22. ↑ 22.0 22.1 Hiroto Nishino, Etsuro Hatano, Satoru Seo, et al. (2018-06). "Real-time Navigation for Liver Surgery Using Projection Mapping With Indocyanine Green Fluorescence: Development of the Novel Medical Imaging Projection System". Annals of Surgery, vol. 267, no. 6, pp. 1134-1140. doi:10.1097/SLA.0000000000002172. https://pubmed.ncbi.nlm.nih.gov/28181939/. Retrieved 2026-10-04.
  23. ↑ Yotam Erel, Daisuke Iwai, Amit H. Bermano (2023). "Neural Projection Mapping Using Reflectance Fields". arXiv:2306.06595; IEEE Transactions on Visualization and Computer Graphics, vol. 29, pp. 4339-4349. https://arxiv.org/abs/2306.06595. Retrieved 2026-10-04.
  24. ↑ Ziyao Wang, Yuqi Li, Wenxing Zheng, Jiaying Chen, Chong Wang (2026-07-20). "FF-ProCams: Feed-Forward Gaussian Splatting for Projector-Camera System". arXiv. https://arxiv.org/abs/2607.17803. Retrieved 2026-10-04.
Retrieved from "https://vrarwiki.com/index.php?title=Projection_mapping&oldid=46431"