Paul Milgram

Paul Milgram is a human factors engineer and Professor Emeritus of Industrial Engineering in the Department of Mechanical and Industrial Engineering at the University of Toronto, where he led the Ergonomics in Teleoperation and Control Laboratory (ETC-Lab). He is best known in virtual reality and augmented reality research for the 1994 paper "A Taxonomy of Mixed Reality Visual Displays", written with Fumio Kishino of the ATR Communication Systems Research Laboratories in Kyoto, which described a "virtuality continuum" running from completely real to completely virtual environments and grouped everything between the two ends as mixed reality. A companion SPIE paper the same year, with Haruo Takemura, Akira Utsumi and Kishino, presented the same framework as the "Reality-Virtuality (RV) continuum". Microsoft's mixed reality developer documentation credits the Milgram and Kishino paper with introducing the term "mixed reality".

Before the taxonomy work, Milgram's laboratory built ARGOS, a monitor-based stereoscopic AR system used in telerobotics research, which Ronald Azuma's 1997 survey of the field showed as an example of monitor-based AR. He also invented the PLATO liquid crystal visual occlusion spectacles, which his company, Translucent Technologies, supplies for perception and driving research.

Reviewed 6 October 2026. Biography, career dates, PLATO and Translucent details, and every cited paper's bibliographic data and content checked against UofT and ETC-Lab pages, the original 1994 papers, Azuma 1997, Skarbez et al. 2021 and Crossref. About review dates.

Education and early career

According to the author biography printed with the 1994 IEICE paper, Milgram received a B.A.Sc. from the University of Toronto in 1970, an M.S.E.E. from the Technion in Israel in 1973, and a Ph.D. from the University of Toronto in 1980. Translucent Technologies, his company, states that his doctoral supervisor in Toronto's Department of Industrial Engineering was John W. Senders, who had used visual occlusion in the 1960s to study the attentional demands of automobile driving. The same biography states that from 1980 to 1982 he was a ZWO Visiting Scientist and a NATO postdoctoral researcher in the Netherlands, studying automobile driving behaviour; his University of Toronto profile places this postdoctoral work at the TNO Institute for Perception in Soesterberg.

He then worked at the National Aerospace Laboratory (NLR) in Amsterdam, as a Senior Research Engineer in Human Engineering according to the 1994 biography (his university profile says senior human factors engineer). The 1994 biography dates that post from 1982 to 1984 and says it covered modelling of aircraft flight crew activity, advanced display concepts, and control loops with human operators in space teleoperation; his university profile describes four years at NLR before he joined Toronto.

Career

University of Toronto

Milgram joined the University of Toronto's Industrial Engineering Department in 1986. By 1994 he was an associate professor, coordinator of the Human Factors Engineering group and cross-appointed to the Department of Psychology. By 2017 and 2018 he was listed as a professor in the Department of Mechanical and Industrial Engineering, director of ETC-Lab, and cross-appointed to the university's Institute of Biomaterials and Biomedical Engineering. In 2017 he gave a keynote at the IAIT2017 conference titled "The Importance of Reference Frames for Characterising Navigation in Information Spaces", which dealt with the range of real and virtual world display-control interactions opened up by mixed reality display technologies. His department lists him as Professor Emeritus of Industrial Engineering, with research interests in human factors of navigation, manipulation and control in 3D environments, human-machine interfaces for teleoperation, human factors in medicine (especially surgery and anaesthesiology), and modelling of attentional workload. The profile describes his research theme as "display, control and navigation issues in 3D (mixed reality) environments", with application domains including surgery, anaesthesiology, telerobotics, air traffic control and automobile driving.

His research leaves abroad included ATR in Kyoto (1993-94), the Centre d'Etudes de la Navigation Aérienne (CENA) in Toulouse (1999-2000), the Universitat Politècnica de Catalunya in Barcelona (2004), the Italian Institute of Technology in Genoa (2010), the Indian Institute of Technology Madras (2011), and the International University of Vietnam National University in Ho Chi Minh City (2017-18).

Stereoscopic augmented reality and ARGOS

ETC-Lab's work combined stereoscopic video of a real scene with superimposed stereoscopic computer graphics. A lab publication list archived in 1997 records papers on the topic from 1989 onward, including "Stereoscopic Video + Superimposed Stereographics: Applications in Teleoperation" (Milgram, David Drascic and Julius J. Grodski, 1989), "A Virtual Stereographic Pointer for a Real Three Dimensional Video World" (Interact '90, 1990), and "Enhancement of 3-D Video Displays By Means of Superimposed Stereographics" (Human Factors Society annual meeting, 1991). Earlier, Milgram and R. van der Horst had published a 1986 paper in the journal Displays on alternating-field stereoscopic displays that used light-scattering liquid crystal spectacles.

The lab's AR system was named ARGOS, short for Augmented Reality through Graphic Overlays on Stereovideo. It was presented as "ARGOS: A Display System for Augmenting Reality" by Drascic, Grodski, Milgram, Ken Ruffo, Peter Wong and Shumin Zhai at the INTERCHI '93 conference in Amsterdam in April 1993. Milgram, Zhai, Drascic and Grodski also presented "Applications of augmented reality for human-robot communication" at the 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93) in Yokohama in July 1993.

In his 1997 survey, Azuma wrote that the University of Toronto researchers used ARGOS, "among other things", to make images easier to understand in difficult viewing conditions, illustrating this with wireframe lines drawn over the interior of a space shuttle bay in orbit. He also wrote that ARGOS "has demonstrated that stereoscopic AR is an easier and more accurate way of doing robot path planning than traditional monoscopic interfaces", and used an external view of ARGOS to illustrate monitor-based AR displays, crediting the images to Drascic and Milgram.

In the 1994 IEICE paper, Milgram and Kishino noted that most published uses of "augmented reality" at the time referred to see-through head-mounted displays, citing work by Steven Feiner and by Tom Caudell and David Mizell among others, while "in the authors' own laboratories" the term had also been applied to monitor-based video displays with computer-generated overlays.

Reality-virtuality continuum

Milgram was an invited researcher at the ATR Communication Systems Research Laboratories in Kyoto in 1993-94, and the SPIE paper notes that it was initiated during that research leave. "A Taxonomy of Mixed Reality Visual Displays" appeared in the December 1994 issue of IEICE Transactions on Information and Systems. It defined a "virtuality continuum" with real environments, "consisting solely of real objects", at one end and virtual environments, "consisting solely of virtual objects", at the other, and called any environment in which real and virtual objects are presented together within a single display a Mixed Reality (MR) environment. The authors gave an operational definition of augmented reality as "any case in which an otherwise real environment is 'augmented' by means of virtual (computer graphic) objects", and proposed the term augmented virtuality (AV) for the converse case, in which a mainly computer-generated world is augmented with video of reality. The paper judged the alternative term "Augmented Virtual Reality", proposed by Cohen in 1993, inadequate for keeping this class distinct from AR, and floated "Hybrid Reality" as a possible label for displays that blend many media. See Reality-virtuality continuum for a fuller account of the framework.

The IEICE paper listed six classes of MR display, from monitor-based video with overlaid graphics through video and optical see-through head-mounted displays to graphic environments into which real objects intrude. Its authors argued that these classes regroup differently depending on whether a system is video- or graphics-based, whether the real world is viewed directly, whether the viewer is egocentric or exocentric, and whether orthoscopic scale must be kept. To separate them, it proposed a three-dimensional taxonomy: Extent of World Knowledge ("how much do we know about the world being displayed?"), Reproduction Fidelity ("how 'realistically' are we able to display it?") and Extent of Presence Metaphor ("what is the extent of the illusion that the observer is present within that world?").

The SPIE paper, "Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum", was published in the proceedings volume Telemanipulator and Telepresence Technologies from the SPIE meeting held in Boston from 31 October to 4 November 1994. It covered the same continuum and taxonomy, but listed seven classes of MR display rather than six, and discussed head-mounted see-through and monitor-based video displays as the two principal forms of AR display.

Later mixed reality research

With David Drascic, Milgram published "Perceptual Issues in Augmented Reality" in the SPIE volume Stereoscopic Displays and Virtual Reality Systems III (San Jose, 1996). The paper argued that all mixed reality systems are limited in their ability to display and control relevant depth cues, so that perceptual biases can interfere with task performance, and identified eighteen issues that pertain to mixed reality in general and to augmented reality in particular. The two followed it with "Perceptual Effects in Aligning Virtual and Real Objects in Augmented Reality Displays" at the 1997 Human Factors and Ergonomics Society annual meeting.

Milgram returned to the classification problem with Herman Colquhoun in "A Taxonomy of Real and Virtual World Display Integration", a chapter in the 1999 Springer book Mixed Reality: Merging Real and Virtual Worlds, edited by Yuichi Ohta and Hideyuki Tamura, and in "A Framework for Relating Head-Mounted Displays to Mixed Reality Displays" at the 1999 Human Factors and Ergonomics Society meeting. At the 2009 ACM Symposium on Virtual Reality Software and Technology (VRST '09) in Kyoto, he contributed "A taxonomy of (real and virtual world) display and control interactions".

His later stereoscopic AR work included studies with Mai Otsuki on stereoscopic pseudo-transparency (IEEE ISMAR 2013) and with Sanaz Ghasemi and Otsuki on using random dot patterns to achieve "X-ray vision" with stereoscopic AR displays (ISMAR 2016 adjunct proceedings).

PLATO spectacles and Translucent Technologies

Milgram developed the PLATO visual occlusion spectacles in the 1980s. According to Translucent Technologies, he and colleagues first built them at the TNO Institute for Perception in Soesterberg to support research on automobile driving behaviour. The name stands for "Portable Liquid crystal Apparatus for Tachistoscopic Occlusion"; the lenses are liquid crystal cells that switch between a transparent state and a translucent, light-scattering state, which prevents the wearer from seeing what lies on the other side while the shutter is closed. He described the device in the paper "A spectacle-mounted liquid-crystal tachistoscope" in Behavior Research Methods, Instruments, & Computers in 1987. In 1990, four years after joining the Toronto faculty, Milgram formed Translucent Technologies with fellow professor Joe Paradi. His 2018 ETC-Lab page listed him as president and CEO of the company, which supplies the spectacles for research on automobile interface design, visual perception, psychomotor coordination, sports training and sports medicine, neurological disorders and cognitive science. The company states that the lenses take approximately 4 milliseconds to open (switch to transparent) and 3 milliseconds to close, with a compensatable delay of about 4 milliseconds.

Influence

The continuum became a standard reference point in AR and VR research. Azuma's 1997 survey described AR as the "middle ground" between virtual environments (completely synthetic) and telepresence (completely real), citing both 1994 Milgram papers. In a 2021 reassessment in Frontiers in Virtual Reality, Richard Skarbez, Missie Smith and Mary C. Whitton wrote that in the following quarter century the work "has been cited thousands of times, cementing it as one of the seminal works in our field". They argued that the continuum is in fact discontinuous, proposed broadening mixed reality to cover any experience in which real and virtual stimuli are presented together in a single percept, and offered a revised taxonomy built on Extent of World Knowledge, Immersion and Coherence.

Industry adopted the vocabulary as well. Microsoft's documentation for Windows Mixed Reality and HoloLens development cites the Milgram and Kishino paper, refers to the "virtuality continuum", and calls the range between physical and digital reality the "mixed reality spectrum".

Selected publications

Year	Title	Co-authors	Venue
1986	Alternating-field stereoscopic displays using light-scattering liquid crystal spectacles	R. van der Horst	Displays
1987	A spectacle-mounted liquid-crystal tachistoscope	None	Behavior Research Methods, Instruments, & Computers
1993	ARGOS: A Display System for Augmenting Reality	David Drascic, Julius Grodski, Ken Ruffo, Peter Wong, Shumin Zhai	INTERCHI '93, Amsterdam
1993	Applications of augmented reality for human-robot communication	Shumin Zhai, David Drascic, Julius Grodski	IROS '93, Yokohama
1994	A Taxonomy of Mixed Reality Visual Displays	Fumio Kishino	IEICE Transactions on Information and Systems
1994	Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum	Haruo Takemura, Akira Utsumi, Fumio Kishino	SPIE vol. 2351
1996	Perceptual Issues in Augmented Reality	David Drascic	SPIE vol. 2653
1999	A Taxonomy of Real and Virtual World Display Integration	Herman Colquhoun	Mixed Reality: Merging Real and Virtual Worlds (Springer)
2009	A taxonomy of (real and virtual world) display and control interactions	None	VRST '09, Kyoto