Binocular disparity

Binocular disparity is the difference in the image location of an object as seen by the left and right eyes, caused by the horizontal separation of the eyes in the head. Because each eye views a scene from a slightly different vantage point, the two retinal images do not match exactly, and the visual system uses this mismatch to recover the relative depth of objects. The perception of solid depth produced from binocular disparity is called stereopsis. The same term is applied in computer vision to the positional difference between matching points in a stereo pair of images, and in stereoscopic 3D technology to the deliberate offset between the images presented to each eye.

Binocular disparity is the working principle behind every stereoscopic display, from the Victorian stereoscope to modern virtual reality and augmented reality head-mounted displays. A VR or AR headset renders or captures the scene twice, from two viewpoints separated by roughly the distance between a person's eyes, and shows one image to each eye. The visual system fuses the pair and interprets the disparity between them as depth, which is what allows a flat pair of screens or a single panel split between the eyes to look three dimensional.

Geometry

The two eyes are offset horizontally by the interpupillary distance (IPD), whose mean and median for the adult population both lie near 63 mm, with the vast majority of adults falling within the range 50 mm to 75 mm. This separation, called the baseline in stereo vision, means that a single point in space projects to slightly different positions on the two retinas. When the eyes fixate on a point, that point falls on corresponding locations in the two retinas and has zero disparity. The set of all points in space that project to corresponding retinal points, and therefore also have zero disparity, is called the horopter.

Objects nearer than the fixation point and objects farther than it produce non-zero disparities of opposite sign. A near object has crossed disparity: its image in the right eye is displaced to the left and its image in the left eye is displaced to the right, so the eyes would have to converge (cross) to fixate it. A far object has uncrossed disparity, requiring the eyes to diverge. The magnitude of disparity grows as an object moves away from the horopter, and disparity decreases as overall viewing distance increases, so the same depth difference produces a smaller disparity when it is farther from the viewer.

There is a limited band of disparity around the horopter within which the two retinal images can still be fused into a single percept; this zone of single binocular vision is called Panum's fusional area. Disparities larger than this band produce double vision (physiological diplopia) rather than fused depth. In computer vision the relation between disparity and distance for a pair of parallel cameras is often written Z = b f / d, where Z is the distance to the point, b is the baseline between the cameras, f is the focal length, and d is the measured disparity, which makes explicit that disparity and distance from the cameras are inversely related.

History

Stereoscopic depth perception from binocular disparity was first demonstrated by the English physicist Charles Wheatstone in a paper, "Contributions to the Physiology of Vision. Part the First. On some remarkable, and hitherto unobserved, Phenomena of Binocular Vision," presented to the Royal Society on 21 June 1838 and published in the Philosophical Transactions of the Royal Society. Wheatstone built a mirror device, the reflecting stereoscope, that presented a separate hand-drawn image to each eye. When the two slightly different drawings were viewed together the brain fused them into a single solid form, proving that the difference between the two eyes' views is itself a cue to depth. Wheatstone received the Royal Medal of the Royal Society in 1840 for the work. The German physiologist Ewald Hering later gave a formal geometric account of horizontal disparity in 1861.

A turning point in understanding came from the Hungarian-born vision scientist Béla Julesz, who in the 1960s introduced the random-dot stereogram, a pair of images made of random dots in which one region is shifted between the two eyes. The shifted region is invisible to either eye alone but appears in clear depth when the pair is fused. Julesz described this work in his 1971 book Foundations of Cyclopean Perception, coining the term "cyclopean" perception for depth that is computed in the brain rather than recognized in either single image. The random-dot stereogram showed that the visual system can extract depth from disparity with no monocular cues, shapes, or outlines present at all.

The problem of deciding which point in the left image corresponds to which point in the right image, the basis for measuring disparity, is known as the correspondence problem. David Marr and Tomaso Poggio published an influential cooperative algorithm for solving it in Science in 1976, "Cooperative computation of stereo disparity," and showed that it could recover depth from Julesz's random-dot stereograms. This line of work links the biology of stereopsis to the stereo-matching methods used in machine vision.

Relation to other depth cues

Binocular disparity is one of several cues the visual system uses to judge distance, and it is the principal cue for stereopsis. It works alongside vergence (the inward or outward rotation of the eyes), and alongside monocular cues that need only one eye, such as relative size, occlusion (interposition), linear perspective, aerial perspective, shading, and parallax from self-motion. Disparity and motion parallax are partly complementary: disparity is most effective for nearer objects and static scenes, while motion parallax contributes more at far distances and in moving scenes, and the two can produce equivalent depth impressions under some conditions.

The strength of disparity as a cue is also limited. Stereoacuity, the smallest depth difference detectable from disparity, declines with viewing distance, so disparity carries little information about the depth of very distant objects, where monocular cues dominate. Research also indicates that the visual system is most sensitive to higher-order structure in the disparity field, such as the local curvature of a surface, rather than to the absolute disparity of isolated points.

Use in virtual and augmented reality

Reproducing binocular disparity is the basic method by which VR and AR headsets create the impression of depth. The headset performs stereoscopic rendering: it renders the virtual scene from two slightly offset virtual cameras, one for each eye, with the offset set to match the user's IPD, and presents the resulting image pair so that each eye sees only its own view. The disparity between the two rendered images drives stereopsis and makes virtual objects appear at definite distances. Because the correct disparity depends on the separation between the eyes, many headsets allow physical or software IPD adjustment; a mismatch between the assumed camera separation and the wearer's actual IPD tends to cause problems with viewing comfort and accurate depth perception.

Disparity-based stereoscopic displays also introduce a well-known limitation. The disparity cue drives the eyes to verge at the simulated distance of a virtual object, but the light reaching the eye comes from a fixed physical display surface, so the eyes accommodate (focus) at that surface rather than at the object's simulated distance. The resulting mismatch is the vergence-accommodation conflict, a source of visual fatigue and discomfort in prolonged use of stereoscopic headsets. Some systems compress or remap disparity into a comfortable range to reduce this fatigue, at the cost of some depth distortion. In machine perception, the same disparity computation is used the other way around: stereo camera pairs measure disparity to estimate the distance of real objects, an approach used for depth sensing and obstacle detection in robotics and in AR passthrough and tracking systems.

See also

• Stereoscopic 3D
• Stereoscopic rendering
• Depth cue
• Vergence-accommodation conflict
• Interpupillary distance
• Binocular overlap