Foveated rendering: Difference between revisions

 

Implementations of foveated rendering are broadly categorized into two types: '''fixed foveated rendering''' (FFR), which assumes the user is always looking at the center of the screen, and '''dynamic (or eye-tracked) foveated rendering''' (ETFR or DFR), which uses integrated [[eye tracking]] hardware to update the high-quality region in real-time to match the user's gaze.<ref name="MetaETFRvsFFR">{{cite web |url=https://developers.meta.com/horizon/blog/save-gpu-with-eye-tracked-foveated-rendering/ |title=Save GPU with Eye Tracked Foveated Rendering}}</ref>

This entire loop must be completed within the frame budget (e.g., under 11.1 ms for a 90 Hz display) to ensure a smooth experience.

* '''Resolution Scaling / Subsampling:''' This is the most common and intuitive method. The peripheral regions are rendered into a smaller off-screen buffer (e.g., at half or quarter resolution) and then upscaled to fit the final display. This directly reduces the number of pixels that need to be processed and shaded.<ref name="GazeContingent2D">{{cite web |url=http://stanford.edu/class/ee367/Winter2017/mehra_sankar_ee367_win17_report.pdf |title=Gaze Contingent Foveated Rendering for 2D Displays}}</ref>

[[Variable Rate Shading]] (VRS) is a hardware feature available on modern GPUs (e.g., [[NVIDIA]] Turing architecture and newer, [[AMD]] RDNA 2 and newer, [[Intel]] Gen11+) that provides fine-grained control over the pixel shading rate.<ref name="TOYF_Paper" /><ref name="OpenXRToolkit">{{cite web |url=https://mbucchia.github.io/OpenXR-Toolkit/fr.html |title=Foveated Rendering - OpenXR Toolkit}}</ref><ref name="microsoft2019">{{cite web |url=https://devblogs.microsoft.com/directx/variable-rate-shading-a-scalpel-in-a-world-of-sledgehammers/ |title=Variable Rate Shading: a scalpel in a world of sledgehammers |publisher=Microsoft DirectX Blog |year=2019}}</ref> It allows a single pixel shader operation to compute the color for a block of pixels, such as a 2x2 or 4x4 block, instead of just a single pixel.<ref name="VarjoAPI">{{cite web |url=https://developer.varjo.com/docs/native/foveated-rendering-api |title=Foveated Rendering - Varjo for Developers}}</ref><ref name="PicoUnrealOpenXR">{{cite web |url=https://developer.picoxr.com/document/unreal-openxr/fixed-foveated-rendering/ |title=Foveated rendering - PICO Unreal OpenXR Plugin}}</ref> The technique supports shading rates from 1×1 (full quality) to 4×4 (coarse, one shade per 16 pixels).

Fixed Foveated Rendering is the most basic implementation of the concept. It operates without any eye-tracking hardware and instead relies on the assumption that a user will predominantly look towards the center of the screen.<ref name="WikipediaFR" /><ref name="JigSpace">{{cite web |url=https://www.jig.com/spatial-computing/foveated-rendering |title=What Is Foveated Rendering? - JigSpace}}</ref> Consequently, FFR systems render a static, high-resolution region in the center of each eye's display, while the quality degrades in fixed concentric rings towards the edges.<ref name="MetaFFRvsETFR">{{cite web |url=https://developers.meta.com/horizon/blog/save-gpu-with-eye-tracked-foveated-rendering/ |title=Save GPU with Eye Tracked Foveated Rendering}}</ref>

Dynamic Foveated Rendering represents the full realization of the concept. It requires a [[head-mounted display]] with integrated eye-tracking cameras to determine the user's precise point of gaze in real-time.<ref name="VarjoWhatIs" /><ref name="WikipediaFR" /> The high-resolution foveal region is then dynamically moved to match this gaze point on a frame-by-frame basis, ensuring that the user is always looking at a fully rendered part of the scene.<ref name="TobiiDFR">{{cite web |url=https://www.tobii.com/resource-center/reports-and-papers/eye-tracking-and-dynamic-foveated-rendering |title=Eye tracking and dynamic foveated rendering - Tobii}}</ref>

* '''Maximum Performance:''' ETFR allows for much more aggressive foveation—a smaller foveal region and a more significant quality reduction in the periphery—resulting in substantially greater performance and power savings.<ref name="VRX_FR_Types" /><ref name="PicoUnrealOpenXR" />

| Moderate. Reduces GPU load by rendering fewer pixels in the periphery (e.g., 26-43% savings reported for Meta Quest).<ref name="QuestProPerformance">{{cite web |url=https://www.uploadvr.com/quest-pro-foveated-rendering-performance/ |title=Quest Pro Foveated Rendering GPU Savings Detailed}}</ref> | Significant. Allows for more aggressive degradation, leading to greater GPU savings (e.g., 33-52% savings reported for Meta Quest Pro).<ref name="VRX_FR_Types" /><ref name="QuestProPerformance" />

* '''Predictive Foveation:''' Some systems attempt to predict the landing point of a saccade based on its initial trajectory and velocity. This allows the rendering system to begin shifting the foveal region to the target destination before the eye movement is complete.<ref name="FoveatedRenderingExplainedReddit" /><ref name="VRX_FR_Types" />

* '''Attention-Aware Foveation:''' This is a cutting-edge research area that aims to model the user's cognitive state of attention. Peripheral visual sensitivity decreases when foveal attention is high.<ref name="AttentionAware" /><ref name="AttentionAwareSIGGRAPH" />

| Meta Quest Pro || Red Matter 2 || Default density || N/A || +33% pixels || 77% more total pixels in optical center<ref name="MetaETFR_Blog" />

| Pimax Crystal || Quad Views || 100% || N/A || 50-100% || More aggressive peripheral reduction<ref name="PimaxDFR" />

| ARM Mali GPU || CircuitVR || 488M cycles || 397M cycles || N/A || 18.6% cycle reduction<ref name="arm2020" />

| NVIDIA GTX 1080 || Shadow Warrior 2 || 60 FPS || N/A || 78 FPS || 30% performance gain<ref name="nvidia2017" />

| PowerGS || 3D Gaussian Splatting || 100% power || N/A || 37% power || 63% power reduction<ref name="vrsplatting2024">{{cite web |url=https://dl.acm.org/doi/10.1145/3728302 |title=VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points |publisher=ACM |year=2024}}</ref>

| 2020 || Neural reconstruction || Facebook Reality Labs || [[DeepFovea]] demonstrated AI-based foveated reconstruction with up to 10-14× pixel count reduction.<ref name="deepfovea2019">{{cite web |url=https://dl.acm.org/doi/10.1145/3306307.3328186 |title=DeepFovea: Neural Reconstruction for Foveated Rendering |publisher=ACM SIGGRAPH |year=2019}}</ref>

| 2021 || Chipset integration || Qualcomm XR2 || Built-in support for foveated rendering and eye tracking in standalone VR chipset.<ref name="WikipediaFR" />

* '''Mobile vs Desktop Performance:''' Mobile GPU architectures see smaller benefits than console/desktop GPUs—Quest Pro achieves 33-45% savings while PSVR2 reaches 72%.<ref name="uploadvr2022" /><ref name="unity2023" />