Rendering

Rendering is the process by which a computer generates an image from a description of a scene: the geometry of objects, their materials and textures, the lights and the position of a virtual camera. In virtual reality and augmented reality, rendering is done in real time by a graphics processing unit (GPU), from two viewpoints (one for each eye), at the refresh rate of a head-mounted display, and from a viewpoint that follows the user's head. Valve engineer Alex Vlachos summarized the load in 2015: at the resolution Valve recommended for the HTC Vive developer kit, a VR application had to shade about 457 million pixels per second, more than a 4K monitor running at 30 Hz.

Because the image has to change "in exactly the way that the image of a real object would change" when the head moves, as Ivan Sutherland put it in 1968, VR rendering adds requirements that ordinary games and films do not have: stereo views (stereoscopic rendering), a strict frame-time budget, pre-distortion of the image for the headset lenses, and techniques such as reprojection and foveated rendering that correct for late or incomplete frames or reduce the amount of work per frame. AR adds a further problem: virtual objects have to be lit and hidden behind real ones in a way that matches the physical scene.

Reviewed 27 September 2026. Checked every claim against its cited source (Valve GDC 2015 slides, Sutherland 1968 paper, Meta ATW/ASW/FFR pages, Unity and ARCore docs, UploadVR, 9to5Mac, Autodesk, Microsoft lifecycle, and the Kajiya, Guenter, Patney and Kerbl paper pages) About review dates.

Definition

Real-time computer graphics is dominated by rasterization. In a rasterizer, objects are built from a mesh of triangles or other polygons; the triangles are projected onto the two-dimensional image and converted into pixels, and each pixel is then assigned a color by shading and texturing. Ray tracing instead works backward from the viewer's eye and follows paths of light through the scene, which captures reflections, refractions and shadows more faithfully. Film studios have long rendered ray-traced frames offline on render farms over days or weeks; rasterization became the standard for interactive graphics because it is much faster, and NVIDIA announced its RTX technology in 2018 with the aim of bringing real-time ray tracing to games.

In "The Rendering Equation", presented at SIGGRAPH 1986, James Kajiya described "an integral equation which generalizes a variety of known rendering algorithms" and a Monte Carlo method for solving it.

Game engines such as Unity supply the renderer for many VR applications and expose VR-specific rendering modes, described below.

History

The head-mounted display that Ivan Sutherland described at the Fall Joint Computer Conference in 1968 already contained a dedicated rendering pipeline. Sutherland wrote that "the fundamental idea behind the three-dimensional display is to present the user with a perspective image which changes as he moves", and that motion of the image mattered more to the illusion than stereo presentation. Because no general-purpose computer of the time was fast enough to produce a flicker-free dynamic picture, his team built a special-purpose digital matrix multiplier, which transformed line endpoints from room coordinates into eye coordinates, and a "clipping divider", which removed lines outside the user's field of view and computed the perspective image. The system could display 3,000 lines at 30 frames per second. Removing hidden lines in real time was judged beyond the team's capability, so the objects it showed were transparent "wire frame" drawings. The display is covered in the article Ivan Sutherland's head-mounted display.

In a February 2013 article, "Latency Mitigation Strategies", John Carmack wrote that absolute delays below approximately 20 milliseconds "are generally imperceptible", and described time warping: re-projecting an already rendered image to match the most recent head orientation just before it is displayed. Oculus described its asynchronous version of the technique in March 2015, the same month Valve presented its own VR rendering methods at the Game Developers Conference, and in November 2016 Oculus added frame extrapolation with Asynchronous Spacewarp.

Requirements of VR rendering

Frame rate and latency

A VR application has to deliver a new frame on every display refresh. The HTC Vive developer edition refreshed at 90 Hz, which leaves 11.11 ms per frame; its low-persistence display lit the panel for only about 2 ms of that interval. Oculus's 2015 guidance likewise told developers to "target a sustained frame rate of 90Hz+", with half rate (45 Hz) as the preferred fallback. On standalone headsets the rates are selectable. Meta Quest 3 apps could request 72, 80, 90 or 120 Hz at launch, with 96 and 100 Hz options added earlier in 2026. According to UploadVR, Horizon OS 2.7, covered on 1 September 2026, lets Quest 3 apps request up to 207 Hz through the standard APIs, and up to 240 Hz in developer mode with display scaling. Meta states that the extended rates are exclusive to Quest 3 and not available on Meta Quest 3S.

Rendering also has to start from a prediction. The head and controller poses used for a frame are predicted forward to the moment its photons will leave the display, and Valve's guidance was to keep that prediction interval as short as possible because accuracy matters more than the total time. The delay between a head movement and the corresponding change on screen is called motion-to-photon latency.

Stereo rendering

Each eye needs its own image from a slightly offset viewpoint. Valve's 2015 talk compared ways to do this on one GPU: running the CPU rendering code twice (rated bad), amplifying geometry in a geometry shader (bad), resubmitting the recorded command buffers for the second eye (good, Valve's solution at the time), and using instancing to draw each object for both eyes in one call (better, with half the API calls). AMD and NVIDIA also offered extensions for splitting the two eyes across two GPUs; Valve reported that AMD's version nearly doubled its frame rate. Unity's current documentation describes three modes. Multi-pass performs a render pass for each eye and has the widest shader compatibility; single-pass instanced renders the scene in one pass using instanced draw calls, which Unity says "significantly decreases CPU usage and slightly decreases GPU usage" compared with multi-pass; and multiview is a variation of single-pass instanced rendering that replaces it on devices that support it.

Lens distortion and render resolution

The lenses of a VR headset magnify the display and bend it with strong radial distortion and color fringing (chromatic aberration). The renderer compensates by pre-warping each frame. In Valve's pipeline, a final warp pass samples the rendered image with three separate sets of texture coordinates, one each for red, green and blue, to correct both the spatial and chromatic distortion. Before the warp, the scene is rendered to an off-screen target larger than the panel. For the HTC Vive, with a 2160 x 1200 framebuffer (1080 x 1200 per eye), Valve recommended about 1.4 times the panel resolution in each dimension, or 1512 x 1680 pixels per eye; Vlachos stressed that each headset design has its own recommended scale, which applications should raise or lower to match the GPU.

Two optimizations follow from the optics. The lenses only show a roughly circular area of each panel, so a "hidden area mesh" supplied by the OpenVR API masks the pixels the user cannot see; Valve measured a 17 percent fill-rate reduction, from 457 to 378 million shaded pixels per second. Trimming the lens distortion mesh itself removed another 15 percent of the warp pass's cost.

Aliasing and image quality

Even at that render scale, each pixel in a 2015 headset covered a much larger angle than on a monitor. Valve calculated about 15.3 pixels per degree for a 110 degree view with the 1.4x scale, against roughly 50 pixels per degree for a 30-inch 2560 x 1600 monitor. Since the user's head never stops moving, aliasing is amplified, and customers described it as "sparkling". Valve treated 4x multisample anti-aliasing as the minimum quality, used 8x when performance allowed, forced 8x anisotropic texture filtering for color and normal maps, and added noise to the framebuffer because gradient banding was more visible than on LCD televisions. Positional tracking adds a content constraint: users can walk up to anything in the tracked volume, so, in Vlachos's words, "if it's in your tracked volume, it must be high fidelity".

Reprojection and frame synthesis

When an application misses a refresh, the display would otherwise repeat the previous image, which the user sees as judder. Asynchronous timewarp (ATW), as described by Oculus chief software architect Michael Antonov in 2015, runs on a separate thread in parallel with rendering and warps the most recent frame to the latest head orientation before every vsync, so a new image is always shown. Oculus documented its limits: orientation-only timewarp ignores head translation, which causes positional judder on near objects; animated objects appear frozen in the warped frames; and reflections and specular highlights, which depend on the eye position, become incorrect. Antonov wrote that ATW "is helpful, but it's not a silver bullet".

Asynchronous Spacewarp (ASW), released by Oculus in November 2016, extended the idea to motion in the scene. When an application cannot hold 90 frames per second on the Oculus Rift, it runs at 45 fps and ASW synthesizes the in-between frames from the motion in previous frames, which Oculus said "almost halves the CPU/GPU time required to produce nearly the same output from the same content". ATW continues to handle head rotation while ASW covers animation and movement within the scene. Oculus presented the feature as a way to support lower minimum-specification PCs.

Foveated rendering

Visual acuity is highest at the fovea and falls off quickly toward the periphery, so a renderer can reduce detail away from the point of gaze. In a 2012 SIGGRAPH Asia paper, Brian Guenter and colleagues at Microsoft Research rendered three nested image layers around the tracked gaze point at progressively lower sampling rates, which reduced the number of pixels shaded by a factor of 10 to 15 and accelerated rendering by a factor of 5 to 6 on a 1920 x 1080 desktop display; they predicted a speedup of 100 times for a 70 degree display with resolution matching foveal acuity. A 2016 NVIDIA study by Anjul Patney and colleagues found that simply blurring the periphery reduced contrast and gave "a sense of tunnel vision"; with contrast enhancement and a temporal anti-aliasing pass, their system reduced the number of shades by up to 70 percent.

Fixed foveated rendering (FFR) needs no eye tracking: it lowers resolution toward the edges of the frame, where lens blur already reduces sharpness. Meta implements FFR on Quest headsets by controlling the resolution of individual render tiles on the GPU, and lists performance gains from about 6.5 to 21 percent depending on the level and on shader complexity; it advises placing high-contrast content such as text near the center of the frame. Eye-tracked foveated rendering moves the high-resolution region to the gaze point. Figures Meta presented to developers for the Meta Quest Pro, reported by UploadVR in October 2022, gave savings of 26 to 36 percent for FFR and 33 to 45 percent for eye-tracked foveated rendering at default resolution, rising to 34 to 43 percent and 36 to 52 percent at 1.5 times the default resolution; UploadVR attributed the gap to the theoretical gains partly to the end-to-end eye-tracking latency, "on the order of 50 milliseconds".

Rendering for augmented reality

In AR, virtual objects are drawn into a view of the real world, so the renderer needs information about the physical environment. For lighting, Google's ARCore Lighting Estimation API analyzes camera images for lighting cues. Its Environmental HDR mode uses machine learning to provide a main directional light that can cast shadows, ambient spherical harmonics for the remaining ambient light, and an HDR cubemap for reflections on shiny objects; a simpler Ambient Intensity mode returns average intensity and color correction values.

Virtual objects also have to be hidden behind real ones. Google calls this occlusion, "accurately rendering a virtual object behind real-world objects", and "paramount to an immersive AR experience". ARCore's Depth API computes a depth map with a depth-from-motion algorithm that compares images taken from different angles as the phone moves, and merges data from a hardware depth sensor such as a time-of-flight sensor when the device has one.

Remote and cloud rendering

When a headset or pair of glasses lacks the GPU power for a scene, the frames can be rendered on another machine and streamed to it. Microsoft offered Azure Remote Rendering, a cloud service for rendering complex 3D models in real time on devices such as HoloLens 2; Microsoft's lifecycle page lists its support start as March 2021, and the service was retired on 30 September 2025.

NVIDIA's CloudXR streams VR and AR applications rendered on RTX GPUs in workstations or the cloud. In March 2026 NVIDIA detailed CloudXR 6.0 support for visionOS 26.4 on the Apple Vision Pro, which 9to5Mac described as the first SDK developed with Apple that allows sharing the user's gaze data; the gaze is used for foveated streaming, which optimizes rendering resolution around where the user is looking, at 4K resolution and 120 Hz. Applications announced with it included Immersive for Autodesk VRED, aimed at automotive design firms. Autodesk announced on 20 May 2026 that the app was available on the App Store for Apple Vision Pro, streaming from RTX-powered systems with foveation through CloudXR.

Consumer services use the same approach. Meta Horizon Hyperscape, shown at Meta Connect 2025, lets Quest 3 and Quest 3S owners scan a room and revisit it as a photorealistic scene; the scenes are reconstructed with Gaussian splatting and cloud streamed from Meta's servers using technology internally codenamed Avalanche.

Research

Research directions in VR rendering include perceptual methods such as foveation, which spend fewer samples where the eye cannot resolve detail, and scene representations other than triangle meshes, as used for Hyperscape. 3D Gaussian splatting, published by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis in ACM Transactions on Graphics in July 2023 and presented at SIGGRAPH 2023, represents a captured scene as a set of 3D Gaussians and renders novel views of it in real time at 1080p resolution. Related methods that use machine learning to represent or synthesize scenes are covered under Neural rendering.

See also

• Stereoscopic rendering
• Foveated rendering
• Fixed foveated rendering
• Variable rate shading
• Reprojection
• Asynchronous timewarp
• Asynchronous Spacewarp
• Motion-to-photon latency
• Level of detail
• Occlusion culling
• Ray tracing
• Neural rendering
• Game engine