Ivan Sutherland's head-mounted 3D display

Ivan Sutherland's head-mounted display worn with the ultrasonic tracking system

Ivan Sutherland's head-mounted three-dimensional display was an experimental head-mounted display and computer graphics system built by Ivan Sutherland together with the student Bob Sproull. Work began at Harvard University around 1966 and continued at the University of Utah, where Sutherland described the completed system in a 1968 conference paper. It is widely regarded as the first head-mounted display driven by computer-generated imagery, and as an early forerunner of both augmented reality and virtual reality.

The mechanical arm that hung from the ceiling to track the wearer's head was nicknamed the Sword of Damocles. That name refers only to the mechanical position sensor, not to the headset or the system as a whole.

The Ultimate Display

The project followed from ideas Sutherland set out in his 1965 essay "The Ultimate Display", published in the proceedings of the IFIP Congress. In it he argued that a display screen should be treated not as a surface for showing pictures but as a window into a mathematical world that the computer maintains. The essay closes with an often-quoted vision: "The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked." The head-mounted display was an early, deliberately modest step toward that goal: rather than fabricating matter, it surrounded the user with a wireframe image that behaved like a real object as the head moved.

Concept

Sutherland framed the system around a simple principle. As he wrote in the paper, the fundamental idea behind a three-dimensional display is to present the user with a perspective image that changes as the user moves, because the image of a real object changes in exactly that way as the observer's head moves. He noted that this dependence on motion, which psychologists call the kinetic depth effect, mattered more to the three-dimensional illusion than stereo presentation by itself, though the system provided stereo as well.

To do this, the apparatus measured the position and orientation of the optical system fastened to the user's head, then computed and drew the appropriate perspective view many times per second. The displayed objects appeared to hang in the space around the user, who could move a few feet, turn around, and tilt the head up or down by about forty degrees while the picture updated accordingly.

Display and optics

The headset carried two tiny cathode-ray tubes, one for each eye, mounted on the optical assembly. Each tube formed a picture about half an inch square with a spot size of roughly six tenths of a thousandth of an inch, giving a resolution comparable to a standard large-tube display of the period. The optics magnified each picture into a virtual image about eighteen inches in front of the corresponding eye, and the user had a field of view of about 40 degrees.

Crucially, the user looked at the synthetic image through half-silvered mirrors in the prisms of the headset. This let the wearer see both the glowing lines from the cathode-ray tubes and the real room at the same time, so the computer graphics could be made to hang in mid-air or to coincide with real surfaces such as walls, a desk top, or the keys of a typewriter. This optical see-through arrangement is why the device is often cited as the first augmented reality display.

The objects shown were transparent wireframe line drawings; the team judged that removing hidden lines for solid, opaque objects in real time was beyond the system's capability. Because the optics and tubes together introduced a pincushion distortion of about three percent, the head-position measurement was designed to be accurate to roughly a tenth of an inch so that displayed material would not appear badly out of place. The two-tube optics presented an independent image to each eye, and the system allowed both a mechanical adjustment for different pupil separations and a software adjustment of the virtual eye separation used for the stereo computation, so the effective interpupillary distance could be matched to the viewer.

Head-position tracking

Two different head-position sensors were built, one mechanical and one ultrasonic, each intended to report the position and orientation of the user's head to the computer. The initial design allowed a working volume of head motion about six feet in diameter and three feet high.

Mechanical sensor (Sword of Damocles)

The mechanical sensor was an arm hanging from the ceiling, free to rotate about a vertical pivot at its ceiling mount. It used two universal joints, one at the top and one at the bottom, together with a sliding center section, to provide the six motions needed to measure both translation and rotation; the position of each joint was reported to the computer by a digital shaft encoder. Sutherland described this mechanical sensor as rather heavy and uncomfortable to use, but built it as a sure method of measuring head position. The imposing overhead structure is what earned the nickname the "Sword of Damocles", which by Sutherland's own account was a joke about the mechanism poised above the user rather than a name for the headset itself. The headset proper was light and rested its weight on the user's head; the overhead arm carried its own weight and tracked motion. The mechanism and its nickname are covered in more detail in the companion article on the Sword of Damocles.

Ultrasonic sensor

The alternative was a continuous-wave ultrasonic sensor. Three transmitters mounted on the head-mounted optical system emitted ultrasound at 37, 38.6, and 40.2 kHz, and four receivers were arranged in a square array on the ceiling, so that phase changes across twelve transmitter-to-receiver paths could be measured. Each path's phase shift was read by the computer as a five-bit number, and the computer counted major phase changes to keep track of motion of more than one wavelength.

Sutherland contrasted this design with the earlier pulsed Lincoln Wand. He chose continuous-wave ultrasound with inexpensive narrow-band transducers to avoid the confusion from pulsed noise, such as the noise made by typewriters, that had troubled the pulsed approach. The trade-off was a wavelength ambiguity: because the wavelength of sound at 40 kHz in air is about a third of an inch, each of the twelve measurements is ambiguous at one-third-inch intervals, which showed up as a constant offset the team called the initialization error. At the time of the 1968 paper the ultrasonic sensor was still being refined and a full report on it was not yet possible.

Computing and graphics hardware

No general-purpose computer of the era was fast enough to generate the dynamic perspective pictures on its own, so the general-purpose computer was used only once per frame, to process the head-position reading and to hold and manipulate the three-dimensional drawing. The heavy work of drawing was handled by special-purpose digital hardware arranged in a pipeline.

The three-dimensional scene was stored in fixed "room" coordinates using homogeneous coordinates, with a fourth scale factor that let points be placed at any distance, including infinitely far away. Each frame, a purpose-built digital matrix multiplier transformed line endpoints from room coordinates into the moving eye coordinate system. It held a four-by-four matrix of 18-bit numbers, delivered a transformed endpoint in about five microseconds, and so performed roughly three million scalar multiplications per second. The transformed endpoints passed to a clipping divider, described in a companion paper by Sproull and Sutherland, which removed the parts of lines outside the user's field of view or behind the observer and performed the division needed for true perspective, processing two endpoints in a little over ten microseconds. Finally a commercial analog line generator drew the lines on the miniature cathode-ray tubes through transistorized deflection amplifiers, taking from about 3 microseconds for the shortest lines to about 36 microseconds for the longest. The equipment could display about 3,000 lines at 30 frames per second, an average of a little over ten microseconds per line.

Development and results

Sutherland reported that he carried out preliminary three-dimensional display experiments during late 1966 and early 1967 at the MIT Lincoln Laboratory, using a relatively crude optical system that presented information to only one eye, with the coordinate transformations and perspective computations performed in software on the TX-2 computer. That early version had no clipping, so any line that ran partly off the screen disappeared entirely, yet the three-dimensional illusion was already convincing: users naturally moved to the viewpoints they wanted, and the size of a displayed cube could be judged by how far the observer had to move to line up with one of its faces.

Later experiments at Harvard and Utah used the improved two-eye optics and the special-purpose graphics hardware. Two kinds of scenes were demonstrated. In one, a "room" with walls labelled N, S, E, and W, plus a labelled ceiling and floor, was displayed around the user, who could look around it by turning the head. In another, a wireframe cube was placed in the center of the working area for the user to examine from any side, and the bond structure of the cyclohexane molecule was also shown. Sutherland noted that the transparent wireframe rendering sometimes allowed ambiguous interpretations, but that observers with stereo vision consistently remarked on the realism of the stereoscopic images.

The work was performed at Harvard University and supported by the Advanced Research Projects Agency (ARPA) under contract SD 265, by the Office of Naval Research under contract ONR 1866(16), and through a long-standing agreement between Bell Telephone Laboratories and the Harvard Computation Laboratory; the early work at MIT Lincoln Laboratory was also supported by ARPA. In the paper's acknowledgments Sutherland credited Robert (Bob) Sproull, then a Harvard senior, who simulated, designed, and built much of the system and debugged the clipping divider, and Quintin Foster, who supervised construction and debugging of the equipment. The ultrasonic head-position sensor was designed and built at the MIT Lincoln Laboratory by Charles Seitz and Stylianos Pezaris of Lincoln Group 23; graduate students Ted Lee and Dan Cohen also contributed, Lee writing programs to display curved surfaces in stereo and Cohen writing the programs used to demonstrate the system.

Significance

The system is generally identified as the first head-mounted display whose imagery was generated and updated by a computer in response to the wearer's head movements, distinguishing it from earlier head-mounted devices that relayed camera or television pictures. Because the optics were see-through, letting computer graphics be overlaid on the real room, it is frequently cited as the earliest augmented reality display as well as a foundational virtual reality experiment. Many of the techniques it required, including hardware-accelerated matrix transformation, clipping, and perspective division in a graphics pipeline, anticipated later real-time computer graphics; Sutherland went on to co-found Evans & Sutherland, a pioneer of graphics hardware. The term "virtual reality" itself was not applied to this work at the time; it was popularized later by Jaron Lanier.

See also

• Sword of Damocles
• Head-mounted display
• Augmented reality
• Virtual reality
• Ivan Sutherland