EyeTap

An EyeTap is an eye-worn device that causes the eye itself to function, in effect, as both a camera and a display. The term and the devices come from Steve Mann, who also calls them Digital Eye Glass. A diverter (a beamsplitter or a double-sided mirror) placed in front of the eye sends light that would otherwise enter the eye into a camera; a wearable computer processes the image; and a display that Mann calls an "aremac" sends rays of light back off the diverter so that they are collinear with the rays that were taken out. Mann's formal requirement, which he calls the EyeTap criterion, is that "each ray of light entering the eye be collinear with a corresponding ray of light entering the device, in at least one mode of operation".

Because the computer sits between the scene and the eye, an EyeTap can add to, remove from or otherwise change what the wearer sees. Mann groups these operations under the name mediated reality, a framework he set out in a 1994 MIT Media Lab technical report, and EyeTap devices were the hardware for his group's research on mediated, augmented and diminished reality. Mann dates the first of his designs, a rough approximation built from a television camera and a miniature cathode-ray tube, to 1978. Guinness World Records lists him as the creator of the first wearable eyeglass camera, in 1980.

Reviewed 4 October 2026. Checked every claim against the cited Mann papers (CARPE 2004, GlassEyes 2012, lab page, 1994 TR, arXiv 2018), Crossref/OpenAlex paper metadata and abstracts, the patent, Guinness, IEEE Spectrum 2013/2025, TechCrunch and the 2012 blog posts and McDonald's France reply. About review dates.

How it works

The University of Toronto's EyeTap Personal Imaging Lab describes the operating principle this way: rays of light that would have entered the eye are instead reflected by the diverter, which is "typically a double-sided mirror or a beamsplitter"; a sensor, typically a CCD camera, collects them; the image is processed; and the aremac (the word "camera" spelled backwards) redisplays it as rays of light that reflect off the diverter again and travel on toward the eye along the original paths. The light the wearer then perceives can be an unaltered copy of the scene or "a computer mediated version of the real world scene". Ideally the aremac also presents the synthetic image at the same focus as the real scene.

In his 2004 CARPE paper Mann restates the design as the co-location of three things: the center of projection of a scene analysis device such as a camera, a view into a scene synthesis device such as an aremac, and the center of projection of the eye itself. His 2001 textbook Intelligent Image Processing gives the principle a full chapter, titled "The EyeTap Principle: Effectively Locating the Camera Inside the Eye as an Alternative to Wearable Camera Systems".

Difference from other head-mounted displays

The lab's description contrasts the EyeTap with the two standard kinds of head-mounted display used for augmented reality. An optical see-through display combines graphics with the real view through partially silvered mirrors, so it "can only add in information": the real world is never blocked out, and virtual content does not occlude real objects except when it is shown much brighter than the scene. A video see-through display shows the wearer camera video, which allows real objects to be hidden completely, but its cameras sit near the eyes rather than at the center of projection of the eye's lens, and the lab notes that this displacement produces a mismatch between what the wearer sees and normal perception. In an EyeTap, the diverter places the camera's center of projection, optically, at that of the eye, so that the mediated part of the view lines up with the unmediated part around it "without any mismatch".

Mann has also described an "illusory transparency" that follows from this geometry: an object blocked by the display hardware remains visible because the device resynthesizes the rays of light that the hardware absorbed. He built a prototype whose eyeglass frames passed directly across the eyes and were "seen through" by computer-controlled laser light.

History

Mann traces the idea to welding. As a child he learned to weld, and he wanted a glass that would darken the bright arc while brightening the dark surroundings; he later wrote that his family built computerized welding helmets as a small business. In a 2013 IEEE Spectrum article he wrote that he began exploring ways to use wearable computers to modify his view in real time "during my youth in the 1970s" and had been "designing, building, and wearing some form of this gear for more than 35 years".

His CARPE paper summarizes the physical evolution: headsets of the 1970s, EyeTaps with the optics outside the eyeglasses in the 1980s, optics built into the glasses in the 1990s, and, around 2000, designs that hid the mediation zones in the frames, lens edges or the cut lines of bifocal lenses. An early version used a helmet-mounted 1.5-inch, 5 kV cathode-ray tube; the miniature CRTs that became available with consumer camcorders made an eyeglass design possible in the early to mid 1980s; and a later 0.6-inch, 6 kV CRT was moved into optics salvaged from an early 1990s television set, with an Internet connection using TCP/IP over the AX.25 amateur packet radio protocol. Mann wrote that an early to mid 1980s eyeglass-based system was later shown at the Smithsonian's National Museum of American History and at SIGGRAPH.

Mann wrote that he took these inventions to MIT in the early 1990s "to start a wearable computing project". His 1994 technical report "Mediated Reality" described a "Wearable Wireless Webcam" with head-mounted display, cameras and wireless communications, and proposed a "personal visual assistant" that remaps the wearer's vision and a "visual memory prosthetic". A 1997 paper in IEEE Computer, "Wearable Computing: A First Step Toward Personal Imaging", described the wearable personal imaging system as one that let him "keep an eye on the screen while walking around".

Generations of Digital Eye Glass

In supplemental material written for his 2012 IEEE Technology and Society Magazine article "Through the Glass, Lightly", Mann numbered his designs as generations of "Glass". He writes the word in the singular, as welders speak of welding glass, and states that generations 2 to 4 were known as "EyeTap Digital Eye Glass".

Generation	Date (per Mann)	Design
Generation-1	1978	An eyeglass with a camera and display integrated into it. Mann's first version used a television camera and a miniature cathode-ray tube over his right eye. Because the camera sat beside the eye, he reported that his adaptation to the offset view persisted after he took the apparatus off.
Generation-2	Not stated	A diverter sends eyeward-bound light into a camera, and the aremac returns rays collinear with the incoming ones, so the eye itself becomes, in effect, the camera and the display.
Generation-3	Not stated	Adds a focus control mechanism so that the eye focuses at the same distance through the glass as it would without it.
Generation-4	Completed 1999	A "pinhole" aremac using a laser light source and a spatial light modulator to give infinite depth of focus, intended to remove the remaining problems when looking at objects in different focal planes, such as a distant object seen through a nearby fence.
Generation-5	Proposed	Holographic video capture and display, in which every ray entering the front of the glass is captured with its direction of arrival.

Mann wrote in 2012 that he had worn the then-current embodiment for 13 years: a strip of aluminum across the forehead with two silicone nose pads, holding an EyeTap with a computer-controlled laser light source in front of his right eye, designed with Chris Aimone. In the same post he said the device is attached so that it "does not come off my skull without special tools".

Research and applications

Mann's lab at the University of Toronto is named the EyeTap Personal Imaging Lab. The table lists peer-reviewed papers on EyeTap devices by Mann and his collaborators.

Year	Publication	Contribution
1999	Mann, Fung and Moncrieff, ACM SIGMOBILE Mobile Computing and Communications Review	Field tests of ENGwear, an electronic news gathering system that uses EyeTap so that the camera shares the eye's point of view, sends and receives images over IP-over-wireless, and records continuously to a circular buffer so the reporter never misses a shot.
2002	Mann and Fung, Presence: Teleoperators and Virtual Environments	Describes the "Reality Mediator", which incorporates an EyeTap, and a tracker for rigid planar patches built on projective geometry and comparametric equations that can selectively augment, diminish or alter part of a scene.
2002	Aimone, Marjan and Mann, International Symposium on Wearable Computers	Combines featureless video motion estimation with a gyroscope for head tracking; on a 1.2 GHz Pentium III wearable computer it registered live video with less than 2 pixels of error (0.3 degrees) at 12 frames per second.
2002	Tang, Aimone, Fung, Marjan and Mann, International Symposium on Mixed and Augmented Reality	Shared mediated reality: one wearer "paints" an environment map with an EyeTap while looking around, and a second wearer browses it by head motion through their own EyeTap.
2004	Mann, ACM CARPE workshop	States the EyeTap criterion and argues that it is required for continuous lifelong capture of personal experience.
2017	Nerkar, Yang, Park, Lu, Papanicolaou and Mann, ACM International Symposium on Wearable Computers	"Open source EyeTap", an open source version of the device aimed at makers; the authors argue that existing augmented reality development platforms are expensive and closed source.

Mediated and diminished reality

The 2002 Presence paper argues that "diminished reality is as important as augmented reality", and that the ability "to not just augment but also deliberately diminish or otherwise alter the visual perception of reality" is what separates mediated reality from augmented reality. The paper states that diminished reality "allows additional information to be inserted without causing the user to experience information overload". The tracking algorithm also takes the camera's automatic gain control into account, estimating motion in tonal as well as spatial coordinates. In the CARPE paper Mann gives other examples: mapping heat to visible light to see in darkness without being blinded by oncoming headlights, and computationally removing advertising from the wearer's view.

A 2018 paper by Mann, Tom Furness, Yu Yuan, Jay Iorio and Zixin Wang adds a "mediality" axis to the 1994 reality-virtuality continuum of Milgram and Kishino. On that chart the EyeTap HDR welding helmet sits near the top middle, combining strong modification of reality with about as much virtual content as a typical augmented reality setup, a region the authors call "augmediated reality".

Lifelogging and sousveillance

Mann's CARPE paper presents the EyeTap as an experience capture system. Because it captures exactly what the wearer sees, he argues, it can serve as a visual memory aid (a "lifeglog", short for lifelong cyborg log), and he uses the word "sousveillance" for the recording of an activity by a participant in it, which he describes as an inverse of surveillance. He also reports using EyeTaps to deliver conference keynotes remotely from the wearer's own point of view, and notes that in conversation people could not tell whether the wearer was looking at the display or at them, because the two are on the same axis.

A patent on the newsgathering application, "Eye-tap for electronic newsgathering, documentary video, photojournalism, and personal safety" (US 6,614,408), was filed by W. Stephen G. Mann on 25 March 1999 and granted on 2 September 2003. Its abstract describes "a camera that captures light passing through the center of a lens of an eye of the user".

Eyestrain and comparison with Google Glass

Mann has argued that devices which put the camera beside the eye are "Generation-1" designs. In his 2012 supplemental paper he classed Google Glass that way, because its camera sits to the right of the eye, and claimed that long-term use of such misaligned devices could cause dizziness, vertigo and "flashback" effects; he presented these as risks from his own experience rather than as results of a user study, noting that no large long-term user population yet existed. In IEEE Spectrum in 2013 he wrote that "the current prototypes of Google Glass position the camera well to the right side of the wearer's right eye" and that "viewing unaligned live video through one eye for an extended time could very well mess up the wearer's neural circuitry". He also criticized the use of fixed-focus lenses to make a display appear farther away, and described instead an arrangement in which the eye behind the mirror can focus at any distance and still see the display clearly, which is what he calls an aremac; ideally, he wrote, a pinhole aremac.

2012 Paris incident

On 1 July 2012 Mann and his family visited the McDonald's restaurant at 140 Avenue des Champs-Élysées in Paris. In a blog post published on 16 July, Mann wrote that an employee first stopped him to ask about the EyeTap, read the letter from his physician and the documentation he was carrying, and let the family order, but that another person later grabbed the device and tried to pull it off his head, that the letter and documents were torn up, and that he was pushed out onto the street. He said the damage stopped the device from overwriting its buffer of processed images, so pictures of the people involved were left in its memory, and he published them.

McDonald's said that McDonald's France learned of the complaint on 16 July and opened an investigation, that the staff members involved all described their "interaction with Dr. Mann" as "polite" and without "a physical altercation", and that staff said they had not damaged any of his possessions. In a 28 July reply that Mann later published, McDonald's France said its inquiry, based on employee and witness statements, "did not verify your version of the events", and that staff had asked him to stop filming to protect the privacy of staff and customers under Article 9 of the French Civil Code. Mann disputed the findings and argued that his computer vision system is a seeing aid in which "no filming is taking place". TechCrunch's report on the denial linked the dispute to privacy concerns about camera glasses, writing that "Google Glass and its eventual clones will make video recording trivial and almost invisible".

Legacy

A 2025 IEEE Spectrum profile of Mann captioned a photo of him wearing the EyeTap as "an extended reality device he invented", and said that the technology he developed "now underlies augmented-reality eyeglasses", including those by Google and Magic Leap.

See also

• Steve Mann
• Diminished reality
• Reality-virtuality continuum
• Optical see-through
• Video see-through
• Smart glasses
• Occlusion
• Privacy in virtual and augmented reality