Lens distortion
More actions
Lens distortion is an optical aberration in which a lens magnifies different parts of an image by different amounts depending on their distance from the optical axis, so that straight lines in the scene appear curved in the image. In virtual reality (VR) and augmented reality (AR) it matters most in head-mounted displays (HMDs), whose magnifying eyepieces give the image of the display panel pincushion distortion. Headset software cancels this by pre-warping each rendered frame with the opposite (barrel) distortion before it reaches the panel, a step usually called distortion correction or predistortion.[1][2]
Distortion changes the shape of the image but not its sharpness. That property is why it can be corrected in software instead of with extra glass. Designers of HMD optics can leave distortion uncorrected, which allows lighter optics, and let the computer-generated image cancel it.[3] Researchers at the University of North Carolina at Chapel Hill described computational predistortion for HMDs in the early 1990s.[3] The 2013 software development kit for the Oculus Rift DK1 left the correction to each application's own pixel shader; on current platforms such as the Oculus PC SDK and OpenXR, the runtime's compositor handles it instead.[2][4][5]
Definition
In the ideal (paraxial) model of a lens, the image of a point is displaced from the optical axis in exact proportion to the point's distance from the axis, so magnification is the same everywhere in the field. Real lenses used at wide angles depart from this. Rolland and Hopkins define distortion as the displacement of the chief rays in the image plane from the positions predicted by first-order optics, and write it as a sum of odd-order terms, ΔR = kR3 + hR5 + higher-order terms, where k and h are the third- and fifth-order coefficients. A lens is often characterized by its percent distortion: the ray displacement divided by the paraxial ray height.[3] Because the magnification is not uniform, distortion also changes brightness across the field, which can be compensated pixel by pixel.[3]
The standard model in photogrammetry and computer vision has two parts. Radial distortion, the larger effect, is caused by the spherical shape of the lens. Decentering distortion, which has both radial and tangential components, comes from lens elements that are decentered or not orthogonal to the optical axis. Alexander Conrady modeled decentering distortion in 1919, and Duane Brown later remodeled it and proposed a radial distortion model; these models have been used in photogrammetry and computer vision for decades.[6] HTC's documentation for its Vive Wave SDK refers developers to the Brown distortion model for the coefficients of its correction polynomial, and Google's Cardboard documentation also describes lens distortion with Brown's model.[7][8]
Radial distortion comes in two forms:
| Type | Appearance | Typical source |
|---|---|---|
| Barrel distortion | Points are pulled toward the center by an amount that grows with distance from the optical axis[1][2] | Short focal length systems[6] |
| Pincushion distortion | Points are pushed away from the center by an amount that grows with distance from the optical axis[1] | Longer focal lengths;[6] the magnifying eyepieces of VR headsets[1][8] |
The two are inverse effects, so an image with pincushion distortion can be corrected with barrel distortion and the reverse.[6] In textbook form the distorted radius is a polynomial of the undistorted radius, such as rd = ru + c1ru3 + c2ru5. With this convention a negative c1 gives barrel distortion and a positive c1 gives pincushion. Steven LaValle notes that higher-order terms are often considered unnecessary in practice.[1] The sign convention depends on whether a formula applies the distortion or compensates for it.[6] The 2013 Oculus SDK wrote its correction as a scaling function f(r) = k0 + k1r2 + k2r4 + k3r6 whose coefficients are all positive for the barrel distortion it applied.[2] A polynomial distortion function generally has no closed-form inverse, so correction software uses approximations or an inverse computed off-line and stored in a table.[1]
Distortion in head-mounted displays
A VR headset places a display panel a short distance in front of each eye and uses wide-angle optics to bring the image into focus as a magnified virtual image.[9] To fill a wide field of view, this lens works with oblique rays far from the paraxial regime. Rolland and Hopkins point out that an HMD also needs a large exit pupil, so the eyes can rotate without vignetting, and that the combination of wide field and large exit pupil makes aberrations hard to balance optically. Correcting every aberration in glass would make the optics too heavy to wear. Their approach was to optimize the lens only for aberrations that reduce sharpness and to pre-warp the computer-generated images so that they cancel the remaining optical warping. Their paper implements this for distortion and notes that lateral chromatic aberration could be corrected the same way.[3] Aberrations that blur the image, such as spherical aberration, coma, astigmatism and field curvature, cannot be corrected this way.[3]
The eyepieces in consumer headsets produce pincushion distortion. The 2013 Oculus SDK documentation for the DK1 states that the Rift's lenses magnify the image to increase the field of view at the cost of pincushion distortion, which the software corrects by introducing a barrel distortion that cancels it.[2] Meta's current Rift documentation describes the same arrangement: the SDK post-processes each rendered view with an equal and opposite barrel distortion so that the two cancel.[4] HTC's Wave SDK documentation also says that VR headsets produce pincushion distortion and that applications apply barrel distortion to make the image look normal.[8]
Uncorrected distortion also affects perception. LaValle notes that static objects would then appear to warp as the user looks around, so the perception of a stationary world fails, and that the incorrect accelerations seen near the periphery may contribute to VR sickness.[1] Warren Robinett and Jannick Rolland named ignoring the optical distortion, along with ignoring differences in interpupillary distance between users, as two common errors in the computational models that generate HMD graphics. They argued that objects keep a constant perceived size, shape and position as the head moves only when the model matches the physical geometry of the headset, including its optics.[10]
The amount of distortion also depends on where the eye is. Rolland and Hopkins warned that their coefficients held only for the eye relief they were computed at, because distortion is "a strong fonction [sic] of the pupil position with respect to the optical system".[3] LaValle likewise notes that asymmetric distortion arises when the eye is not centered on the lens, that the distortion varies and becomes asymmetric as the pupil moves along a spherical arc when the eye rotates, and that it changes when the distance between the lens and the screen is adjusted.[1] A correction calibrated for one nominal eye position therefore leaves residual warping that changes with gaze, an artifact known as pupil swim.[11]
Correction methods
Software correction, also called predistortion or pre-warping, moves every point of the rendered image in the opposite direction to the lens distortion. When the lens then distorts the pre-warped image, each point lands where an ideal lens would have placed it.[3] Several ways of doing this have been used:
| Method | How it works | Notes |
|---|---|---|
| Vertex warping | Polygon vertices are moved to their pre-distorted screen positions before rasterization | Described by Rolland and Hopkins (1993) for the Pixel-Planes 5 graphics computer; large polygons must be subdivided, because their straight edges would otherwise appear curved[3] |
| Texture mapping | The undistorted frame is rendered, then used as a texture on a grid whose vertices have been pre-distorted | Proposed by Watson and Hodges (1995); the texture hardware's built-in filtering handles predistortion aliasing[12] |
| Pixel shader post-process | A full-screen shader computes, for each output pixel, the position in the rendered frame to sample from | Used in the 2013 Oculus SDK for the DK1;[2] described by LaValle as "distortion shading"[1] |
| Distortion (warp) mesh | A precomputed mesh stores the distortion in its texture coordinates, and the GPU interpolates between vertices | Adopted by Oculus in the SDK 0.3.1 preview (2014);[13] used by Valve's SteamVR[14] and the Google Cardboard SDK[15] |
| In-camera (object-space) correction | The scene is sampled directly on a barrel-distorted grid, for example with a ray tracer, so no resampling step is needed | Demonstrated by Pohl, Johnson and Bolkart (2013)[9] |
Watson and Hodges also considered correcting the distortion with additional optics that reverse it. They noted that such optics can be expensive and suit only one specific HMD, and that computing the correction in software with incremental methods or a lookup table was impractical because of an I/O bottleneck. Their alternative relied on then-new texture-mapping hardware that could accept undistorted images at interactive rates.[12]
Render target size and wasted pixels
Barrel pre-warping pulls pixels in toward the center of each eye's image. If the scene is rendered at panel resolution and then warped, black borders appear where no image data exists and part of the field of view is lost. The 2013 Oculus SDK therefore rendered to a larger intermediate buffer. Its example: for the Rift's 1280 x 800 panel, enlarging the input image by 25 percent required a 1600 x 1000 buffer, 1.56 times as many pixels.[2] Pohl and colleagues measured that 15.2 percent of the pre-warped image area was unused.[9]
Valve's Alex Vlachos gave figures for the HTC Vive Developer Edition at the 2015 Game Developers Conference. Its 2160 x 1200 framebuffer (1080 x 1200 per eye) was rendered off-screen at about 1.4 times each dimension, or 1512 x 1680 per eye. Vlachos said that 1.4 was a recommendation specific to that headset, since each HMD design has its own scale factor based on its optics and panels.[14] Because the lens distortion is radially symmetric, the user sees only a roughly circular area of each panel. SteamVR supplies a "hidden area mesh" that masks the pixels the user cannot see, which Valve said reduced fill rate by 17 percent. Shrinking the warp mesh to the visible area culled a further 15 percent of the pixels from the distortion pass.[14]
GPU vendors have also changed how scenes are rendered to account for the warp. NVIDIA's Lens Matched Shading, part of VRWorks for Pascal-generation GPUs, renders to a surface that "more closely approximates the lens corrected image", which avoids rendering many pixels that would otherwise be discarded before the image is output to the headset.[16] In Road to VR's 2016 report, NVIDIA said the technique could give up to a 50 percent increase in throughput available for pixel shading.[17] NVIDIA describes the technique as an improvement on its earlier Multi-res shading.[16]
Image quality
Warping a finished image resamples it, and interpolation between pixels softens fine detail. Pohl, Johnson and Bolkart, working at Intel and Saarland University, compared correction methods using a ray tracer and the distortion parameters from the Oculus Rift DK1 SDK. Rated with the SSIM image quality index against their in-camera correction (1.0), bilinear warping scored 0.94 and bicubic warping 0.95. Rendering at a higher resolution before warping also recovered sharpness.[9] In their tests, rendering onto a distortion mesh converged toward the result of bilinear pixel-shader warping as the triangle count rose.[9]
Chromatic aberration correction
A lens refracts different wavelengths by different amounts, so the red, green and blue parts of an image are magnified slightly differently. This lateral chromatic aberration shows as colored fringes that grow toward the edge of the lens. Like distortion, it warps the image without blurring it, so it can be corrected computationally.[3][2] The usual method scales each color channel radially by a different amount during the distortion pass.[1][2] Valve's SteamVR warp pass uses three sets of texture coordinates, one per color channel, to correct spatial and chromatic distortion together,[14] and the OpenVR function IVRSystem::ComputeDistortion returns separate red, green and blue coordinates for each point of the distortion map.[18]
Per-channel correction cannot remove all color fringing. The 2013 Oculus documentation explains that each color channel of an LCD panel spans a range of wavelengths, so aligning the peaks of the three channels still leaves aberration within each channel. In optical design, chromatic aberration across a wide range of wavelengths is typically managed by combining optical elements, for example in achromatic doublets.[2] Pohl and colleagues proposed correcting chromatic aberration with three distortion meshes, one per color channel, blended additively.[9]
History
An early example of cancelling a viewer lens's distortion in the image itself comes from wide-angle stereoscopic photography. Eric Howlett, whose LEEP wide-angle stereoscopic optics were later used in early VR headsets, filed a patent in 1980 for a wide-angle photography system. His camera lens deliberately introduced "a large and operative amount of positive distortion and lateral chromatism", and the uncorrected magnifying lenses of the viewer then restored straight lines and neutralized the color fringing.[19] The VPL EyePhone and the Virtual Research Flight Helmet both used LEEP optics. Rolland and Hopkins computed their distortion at 17.88 percent and 19.33 percent respectively, at a radial screen distance of 22.38 mm and an eye relief of 25 mm.[3]
| Date | Development |
|---|---|
| 1980-1983 | Eric Howlett's patent on complementary camera and viewer distortion is filed (25 March 1980) and published (27 September 1983)[19] |
| 1992 | Robinett and Rolland publish a computational model of HMD optics that includes lens distortion, with parameters for the VPL EyePhone[10] |
| 1993 | Rolland and Hopkins describe computational predistortion by vertex warping on the Pixel-Planes 5 system at UNC Chapel Hill[3] |
| 1995 | Watson and Hodges present texture-mapped distortion correction at VRAIS '95[12] |
| 2013 | The Oculus SDK (version 0.2.5 dated 9 October 2013) corrects the Rift DK1's pincushion distortion with a barrel-distortion pixel shader that applications ran themselves[2] |
| 15 April 2014 | Oculus SDK 0.3.1 preview adds "SDK distortion rendering", models distortion with spline curves instead of polynomials, tailors it to user profile settings, and switches client-side distortion from a pixel shader to a mesh[13] |
| 2015 | Valve describes SteamVR's warp mesh, per-channel chromatic correction and hidden area mesh at GDC[14] |
| May 2016 | NVIDIA details Lens Matched Shading for Pascal GPUs[17] |
| 2022 | Researchers at Meta publish the first user study of perceptual requirements for eye-tracked distortion correction[11] |
Platform implementations
With SDK 0.2.5, distortion correction ran as a post-process in the application's own rendering code.[2] With the 0.3.1 SDK in 2014, Oculus recommended letting the SDK handle distortion, frame timing and buffer swap after the application rendered its stereo views.[13] In the current Oculus SDK for the Rift, distortion correction runs automatically in the compositor process, which applies timewarp, distortion and chromatic aberration correction to each layer separately before blending them.[4] The OpenXR specification follows the same model: composition layers let applications hand final composition to a runtime-supplied compositor, so that "details such as frame-rate interpolation and distortion correction can be performed by the runtime".[5]
OpenVR exposes the headset's distortion through ComputeDistortion, which returns, for a point in one eye's viewport, the source render-target coordinates for each color channel, for use in a distortion map.[18] HTC's Vive Wave SDK uses a polynomial with coefficients K0 to K6 and refers developers to the Brown distortion model for their values; negative values produce barrel distortion and positive values pincushion distortion.[8]
Google Cardboard had to support many third-party viewers with different lenses. Manufacturers generate a viewer profile, shared with phones through a QR code, that records the viewer's optical and mechanical parameters, including lens distortion coefficients.[20] Google says the Cardboard SDKs approximate the ideal distortion model with two coefficients, k1 and k2, which manufacturers can enter if known or establish empirically, adjusting them until "all angles appear to be 90° and all the lines are straight" through the lenses, both in the center of the view and in the periphery.[7] The SDK's lens distortion module then produces a distortion mesh for each eye from those parameters.[15]
Augmented reality and passthrough
Optical see-through AR displays need the same correction, often with harder geometry. Compact, wide-field near-eye designs can produce gaze-contingent, non-linear distortions that explicit geometric models find hard to represent. Hiroi, Someya and Itoh proposed a neural distortion field, a neural network that takes a spatial position and gaze direction as input and outputs the corresponding display pixel coordinate. On an AR near-eye display with a 90 degree field of view, it achieved a median error of about 3.23 pixels (5.8 arcmin) from eight training viewpoints and was more accurate than non-linear polynomial fitting, especially near the center of the field.[21] Xiao and colleagues described correction methods for VR and AR eyewear displays with asymmetrical and nonlinear distortion. Their methods combine image segmentation with linear approximation to build a pixel-mapping table for real-time predistortion.[22]
Video passthrough headsets have a second source of distortion: the outward-facing cameras. Camera calibration methods can model the cameras' own lens distortion; Zhang's 2000 planar-pattern calibration technique, for example, models radial lens distortion.[23] Lens distortion is not the only geometric error in passthrough, though. Researchers at the U.S. Food and Drug Administration's Center for Devices and Radiological Health measured geometric distortion on single-camera and dual-camera video see-through HMDs. They found that camera misalignment relative to the eye positions was the primary cause. Distortion correction partly reduced the error in a 2D plane, but correcting it at multiple depths in 3D space remained a challenge.[24]
Research
Because distortion changes with pupil position, a fixed correction is exact for only one eye position. Eye-tracked, or dynamic, distortion correction updates the pre-warp using the tracked position of the eye. Phillip Guan, Olivier Mercier, Michael Shvartsman and Douglas Lanman of Meta built a VR display simulator that reproduces the gaze-contingent distortion of any viewing optic on a high-speed television with shutter glasses, so that optics can be studied without fabricating them. They used it for what they describe as the first user study of the perceptual requirements for eye-tracked optical distortion correction, and proposed a binocular distortion metric that agreed with key trends in the study.[11]
Other work has addressed the mathematics of the inverse. Drap and Lefèvre derived an exact formula for the coefficients of the inverse of a polynomial radial distortion model. The inverse is itself a polynomial, which avoids iterative solutions.[6] Graphics-hardware correction has also been studied outside VR. Bax and Shahidi showed that correction using commodity texture-mapping hardware runs at video frame rates and is faster than using the main processor, and that a polar mesh is more accurate than a conventional grid.[25]
See also
References
- ↑ 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 Steven M. LaValle (2020-11-11). "Correcting Optical Distortions (Section 7.3)". Virtual Reality (online book). https://lavalle.pl/vr/node211.html. Retrieved 2026-09-29.
- ↑ 2.00 2.01 2.02 2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11 Michael Antonov, Nate Mitchell, Andrew Reisse, Lee Cooper, Steve LaValle, Max Katsev (2013-10-09). "Oculus VR SDK Overview, SDK Version 0.2.5". Oculus VR developer documentation (archived copy). Oculus VR. https://web.archive.org/web/2014/http://static.oculusvr.com/sdk-downloads/documents/Oculus_SDK_Overview.pdf. Retrieved 2026-09-29.
- ↑ 3.00 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09 3.10 3.11 Jannick P. Rolland, Terry Hopkins (1993). "A Method of Computational Correction for Optical Distortion in Head-Mounted Displays". University of North Carolina at Chapel Hill, Department of Computer Science, Technical Report TR93-045. http://www.cs.unc.edu/techreports/93-045.pdf. Retrieved 2026-09-29.
- ↑ 4.0 4.1 4.2 "Rendering to the Rift". Meta Horizon OS Developers. Meta. https://developers.meta.com/horizon/documentation/native/pc/dg-render/. Retrieved 2026-09-29.
- ↑ 5.0 5.1 "The OpenXR Specification, version 1.1 (section 10.6.3, Composition Layer Types)". Khronos Registry. The Khronos Group. https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html. Retrieved 2026-09-29.
- ↑ 6.0 6.1 6.2 6.3 6.4 6.5 Pierre Drap, Julien Lefèvre (2016). "An Exact Formula for Calculating Inverse Radial Lens Distortions". Sensors, vol. 16, no. 6, article 807. https://doi.org/10.3390/s16060807. Retrieved 2026-09-29.
- ↑ 7.0 7.1 "Enter physical viewer parameters". Cardboard Manufacturer Help. Google. https://support.google.com/cardboard/manufacturers/answer/6324808?hl=en. Retrieved 2026-09-29.
- ↑ 8.0 8.1 8.2 8.3 "Distortion Correction Theory". VIVE Wave VR 6.2.0 documentation. HTC. https://hub.vive.com/storage/app/doc/en-us/DistortionCorrectionTheory.html. Retrieved 2026-09-29.
- ↑ 9.0 9.1 9.2 9.3 9.4 9.5 Daniel Pohl, Gregory S. Johnson, Timo Bolkart (2013). "Improved Pre-Warping for Wide Angle, Head Mounted Displays". Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology (VRST '13), pp. 259-262. doi:10.1145/2503713.2503752. https://www.qwrt.de/pdf/Improved-Pre-Warping-for-Wide-Angle-Head-Mounted-Displays.pdf. Retrieved 2026-09-29.
- ↑ 10.0 10.1 Warren Robinett, Jannick P. Rolland (1992). "A Computational Model for the Stereoscopic Optics of a Head-Mounted Display". Presence: Teleoperators and Virtual Environments, vol. 1, no. 1, pp. 45-62. https://doi.org/10.1162/pres.1992.1.1.45. Retrieved 2026-09-29.
- ↑ 11.0 11.1 11.2 Phillip Guan, Olivier Mercier, Michael Shvartsman, Douglas Lanman (2022). "Perceptual Requirements for Eye-Tracked Distortion Correction in VR". ACM SIGGRAPH 2022 Conference Proceedings. https://doi.org/10.1145/3528233.3530699. Retrieved 2026-09-29.
- ↑ 12.0 12.1 12.2 Benjamin A. Watson, Larry F. Hodges (1995). "Using texture maps to correct for optical distortion in head-mounted displays". Proceedings of the Virtual Reality Annual International Symposium '95 (VRAIS '95), IEEE, pp. 172-178. https://doi.org/10.1109/VRAIS.1995.512493. Retrieved 2026-09-29.
- ↑ 13.0 13.1 13.2 Ben Lang (2014-04-15). "New Oculus Rift SDK v0.3.1 Preview: John Carmack's Timewarp, Improved Distortion, and More". Road to VR. https://www.roadtovr.com/oculus-rift-sdk-v0-3-1-preview-distortion-john-carmack-timewarp/. Retrieved 2026-09-29.
- ↑ 14.0 14.1 14.2 14.3 14.4 Alex Vlachos (2015). "Advanced VR Rendering". Game Developers Conference 2015 (presentation slides). Valve. https://media.steampowered.com/apps/valve/2015/Alex_Vlachos_Advanced_VR_Rendering_GDC2015.pdf. Retrieved 2026-09-29.
- ↑ 15.0 15.1 "Lens Distortion (C API Reference for Cardboard SDK)". Google for Developers. Google. https://developers.google.com/cardboard/reference/c/group/lens-distortion. Retrieved 2026-09-29.
- ↑ 16.0 16.1 "VRWorks - Lens Matched Shading". NVIDIA Developer. NVIDIA. https://developer.nvidia.com/vrworks/graphics/lensmatchedshading. Retrieved 2026-09-29.
- ↑ 17.0 17.1 Ben Lang (2016-05-17). "NVIDIA Explains Pascal's 'Lens Matched Shading' for More Efficient VR Rendering". Road to VR. https://www.roadtovr.com/nvidia-explains-pascal-simultaneous-multi-projection-lens-matched-shading-for-vr/. Retrieved 2026-09-29.
- ↑ 18.0 18.1 "IVRSystem::ComputeDistortion". OpenVR wiki (GitHub). Valve. https://github.com/ValveSoftware/openvr/wiki/IVRSystem::ComputeDistortion. Retrieved 2026-09-29.
- ↑ 19.0 19.1 Eric M. Howlett (1983-09-27). "US4406532A - Wide angle color photography method and system". Google Patents. https://patents.google.com/patent/US4406532A/en. Retrieved 2026-09-29.
- ↑ "Overview of generating a QR viewer profile". Cardboard Manufacturer Help. Google. https://support.google.com/cardboard/manufacturers/answer/6321873?hl=en. Retrieved 2026-09-29.
- ↑ Yuichi Hiroi, Kiyosato Someya, Yuta Itoh (2022). "Neural distortion fields for spatial calibration of wide field-of-view near-eye displays". Optics Express, vol. 30, no. 22, p. 40628. https://doi.org/10.1364/OE.472288. Retrieved 2026-09-29.
- ↑ Xue Xiao, Lin Zhang, Xiao Lin, Jianying Hao, Jinliang Zang, Xiaodi Tan (2019). "Computational Correction Method for Optical Distortion in Eyewear Displays". SID Symposium Digest of Technical Papers, vol. 50, no. S1, pp. 516-519. https://doi.org/10.1002/sdtp.13548. Retrieved 2026-09-29.
- ↑ Zhengyou Zhang (2000). "A flexible new technique for camera calibration". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334. https://doi.org/10.1109/34.888718. Retrieved 2026-09-29.
- ↑ Chumin Zhao, Ryan Beams (2024). "Geometric distortion on video see-through head-mounted displays". Journal of the Society for Information Display, vol. 32, no. 5, pp. 184-193. https://doi.org/10.1002/jsid.1282. Retrieved 2026-09-29.
- ↑ Michael R. Bax, Ramin Shahidi (2014). "Real-time lens distortion correction: speed, accuracy and efficiency". Optical Engineering, vol. 53, no. 11, article 113103. https://doi.org/10.1117/1.OE.53.11.113103. Retrieved 2026-09-29.