Motion interpolation
More actions
Motion interpolation, also called frame interpolation or video frame interpolation, is the generation of new intermediate frames between existing frames of a video or rendered image sequence, using an estimate of how image content moves from one frame to the next. It raises the frame rate of the displayed sequence above the rate at which frames were captured or rendered. In the description of Niklaus, Mai and Liu (ICCV 2017), standard methods "first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion".[1] The technique is used for broadcast frame rate conversion, in the "motion smoothing" settings of flat-panel televisions, and in the frame generation features of PC graphics cards.[2][3][4]
In virtual reality, frame synthesis is used to keep a head-mounted display fed at its full refresh rate when an application cannot render fast enough. True interpolation needs the frame that comes after the generated one, and waiting for that frame adds latency.[5] The main PC VR runtimes extrapolate new frames from past frames instead. Valve wrote in 2018 that television-style interpolation "adds latency" and is "definitely not the right way to go in VR", and its Motion Smoothing feature extrapolates a new frame from the last two delivered frames.[6] Oculus' Asynchronous Spacewarp also generates extrapolated frames from previous frames.[7] UploadVR's explainer on VR frame synthesis covers these extrapolation features together with reprojection techniques such as timewarp.[8]
Definition and terminology
Motion interpolation takes two or more existing frames and computes one or more frames that represent moments between them. The authors of Super SloMo (CVPR 2018) state the task as: "Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences."[9] When the frames are built by warping along estimated motion vectors, the research literature calls it motion-compensated frame interpolation.[10]
Three related operations are distinguished in VR and real-time graphics:
| Operation | Input frames | Output | Example |
|---|---|---|---|
| Interpolation | A frame before and a frame after the target time | An in-between frame | Television motion smoothing; NVIDIA DLSS Frame Generation[6][4] |
| Extrapolation | Past frames only | A frame predicted forward in time | Asynchronous Spacewarp; SteamVR Motion Smoothing[7][6] |
| Reprojection (time warp) | The latest rendered frame plus newer head-tracking data | The same frame re-warped to the current head pose | Timewarp and Asynchronous Timewarp[11] |
Reprojection by itself does not account for animation or object motion: UploadVR's 2019 explainer notes that timewarp and reprojection, as then implemented, "only account for rotational tracking" and not for positional head movement or for objects moving in the scene, which is the gap ASW and Motion Smoothing were built to fill.[8]
How it works
Motion estimation
The first step is to estimate a motion vector field that says where each block or pixel of one frame has moved in the next. A 1987 BBC Research Department report by G. A. Thomas reviewed four existing motion vector measurement techniques, proposed extensions to a method based on phase correlation, and investigated applications including temporal standards conversion.[2] In 1993 Gerard de Haan and colleagues published the 3-D recursive search block matcher, which uses "only eight candidate vectors per block" and was evaluated "with criteria relevant for the field rate conversion application".[12] Dense optical flow estimation, which assigns a motion vector to every pixel, is the other common approach; the Middlebury optical flow benchmark of Baker and colleagues includes high frame rate video specifically "used to study interpolation error" and reports frame interpolation error as one of its measures.[13]
Frame synthesis and occlusion
Once motion vectors are available, the system warps pixels from the input frames along their estimated paths to the target time and blends the results. The hard cases are regions that are visible in only one input frame. When an object moves, it uncovers background that was hidden behind it (disocclusion), and the warped frame has no data for that area. Super SloMo handles this by predicting "soft visibility maps" and excluding "the contribution of occluded pixels to the interpolated intermediate frame to avoid artifacts".[9] An extrapolating system has less to work with, because it has no later frame showing what was uncovered. Oculus wrote that ASW "doesn't know what's going to be there, so the world behind will be stretched to fill the void", producing what it called object disocclusion trails.[7]
Learned methods
Many recent methods train neural networks to perform part or all of the task. Niklaus, Mai and Liu replaced explicit flow estimation with a convolutional network that predicts pairs of one-dimensional kernels for every pixel and convolves the input frames with them.[1] Super SloMo computes bi-directional optical flow with a U-Net, refines it, and can produce any number of intermediate frames because none of its learned parameters are time-dependent; it was trained on 1,132 video clips at 240 frames per second containing 300,000 frames.[9] RIFE (ECCV 2022) estimates intermediate flow directly with a network called IFNet and its authors report it as 4 to 27 times faster than SuperSlomo and DAIN.[14] FILM (ECCV 2022) targets large motion between the two input images and uses a single network trained "from frames alone".[15]
Real-time renderers can also feed data from the game engine into the process. NVIDIA's DLSS Frame Generation network takes four inputs, "current and prior game frames, an optical flow field generated by Ada's Optical Flow Accelerator, and game engine data such as motion vectors and depth".[4] Meta's Application SpaceWarp on Quest headsets likewise requires the application to render "a motion vector buffer and a depth buffer, in addition to the standard eye buffer".[16]
Artifacts
Imperfect frame synthesis shows up as visible defects in the generated frames. Oculus listed the typical ASW artifacts as rapid brightness changes such as lightning or strobes, object disocclusion trails, repeated patterns under fast motion, and head-locked elements that move too fast to track.[7] Microsoft's guide to motion reprojection for Windows Mixed Reality headsets warned that sharp contrast edges or text, "especially on in-game HUDs or menus, may look temporarily warped or distorted because of disocclusion".[17]
History
| Year | Development |
|---|---|
| 1987 | BBC Research Department report studies phase-correlation motion measurement for temporal standards conversion.[2] |
| 1993 | De Haan et al. publish 3-D recursive search block matching for field rate conversion.[12] |
| 1997 | Mark, McMillan and Bishop present post-rendering 3D warping, which derives extra frames from rendered frames and their depth buffers.[18] |
| 2015 | Sony describes a PlayStation VR mode that renders at 60 Hz and displays at 120 Hz through reprojection.[19] |
| 2016 | Oculus releases Asynchronous Spacewarp with Oculus runtime 1.10.[7] |
| 2017 | Adaptive separable convolution (Niklaus et al.) is presented at ICCV.[1] |
| 2018 | Super SloMo is presented at CVPR; Valve releases SteamVR Motion Smoothing; Tom Cruise and Christopher McQuarrie ask viewers to switch off TV motion smoothing.[9][6][3] |
| 2019 | The UHD Alliance introduces Filmmaker Mode, which disables TV motion smoothing.[20] |
| 2021 | Virtual Desktop adds Synchronous Spacewarp; Meta introduces Application SpaceWarp for Quest.[21][16] |
| 2022 | NVIDIA announces DLSS 3 with Frame Generation for GeForce RTX 40 Series GPUs; RIFE and FILM are presented at ECCV.[4][14][15] |
| 2023 | AMD FSR 3 with frame generation debuts in its first games.[22] |
Television and video
Television makers use motion interpolation to reduce blur in fast motion. Variety described the feature as "designed to reduce blur in high-motion video, like live sports", with sets inserting frames "that didn't exist in the source video to smooth out the action". Brand names include Sony's MotionFlow, Samsung's Auto Motion Plus and LG's TruMotion, and Variety reported that the feature "is typically enabled on most HDTVs" by default.[3]
Variety noted that motion smoothing is also known as "the soap opera effect".[3] Fortune wrote that movies are typically shot at 24 frames per second and rely on motion blur, and that motion smoothing can make a movie seem "artificially hyperreal".[20] In a video posted in December 2018, Tom Cruise said the feature "makes most movies look like they were shot on high-speed video, rather than on film", and Christopher McQuarrie told viewers that with a modern high-definition television there was "a good chance you're not watching movies the way the filmmakers intended".[3] In August 2019 the UHD Alliance, working with directors including Martin Scorsese, Christopher Nolan and Rian Johnson, introduced Filmmaker Mode.[20] The UHD Alliance states that Filmmaker Mode "disables certain post-processing features such as motion smoothing, sharpening, noise reduction, and others".[23]
Real-time graphics and games
GPU vendors brought interpolation to PC games in 2022 and 2023. NVIDIA's DLSS 3, announced on 20 September 2022, combines DLSS Super Resolution, DLSS Frame Generation and NVIDIA Reflex. For each pixel the Frame Generation network "decides how to use information from the game motion vectors, the optical flow field, and the sequential game frames to create intermediate frames". NVIDIA stated that DLSS 3 is powered by the fourth-generation Tensor Cores and Optical Flow Accelerator of its Ada Lovelace architecture, which is used in GeForce RTX 40 Series graphics cards.[4] According to AMD's GPUOpen documentation, FidelityFX Super Resolution 3 adds frame generation through two new technologies, Frame Interpolation and Optical Flow, the latter enhanced from AMD Fluid Motion Frames.[24] The first FSR 3 game integrations shipped on 29 September 2023 as patches for Forspoken and Immortals of Aveum.[22]
Academic work has looked for ways around the latency cost. The authors of ExtraNet (ACM Transactions on Graphics, 2021) wrote that temporal supersampling methods that produce more frames "on the fly" were still not practically available, "mainly due to both its own computational cost and the latency introduced by interpolating frames from the future". ExtraNet instead predicts shading on an extrapolated frame from rendered geometry buffers and motion vectors, and its authors report a 1.5x to nearly 2x increase in frame rate.[5]
Applications in VR and AR
Why VR extrapolates
A VR headset must show an image that matches the user's current head pose. Van Waveren's 2016 paper on asynchronous time warp, written "to help create a true sense of presence", describes warping a rendered stereo pair to the latest head tracking data to "significantly reduce the motion-to-photon delay", and notes that when run asynchronously the warp "can be used to increase the perceived frame rate and to smooth out inconsistent frame rates".[11] Interpolating between two rendered frames would require waiting for the later one, the latency penalty Valve cited when it chose extrapolation for SteamVR.[6]
The idea of producing extra VR frames by warping rendered images predates consumer headsets. Mark, McMillan and Bishop's 1997 paper, classified under the ACM computing category for virtual reality, observed that "in typical immersive applications, the viewpoint changes gradually, so that adjacent frames are very similar", and proposed generating most frames by image warping "to extrapolate from nearby conventionally rendered frame(s)". They reported that re-rendering from previously computed views allows "an order-of-magnitude increase in apparent frame rate" and can also compensate for system latency, and they warped two reference images to reduce occlusion artifacts.[18]
Consumer implementations
In the auto-toggled systems described by UploadVR, the compositor turns frame synthesis on when the frame rate has been low for more than a few seconds, forces the application to half frame rate, and synthesizes every second frame.[8]
| Technique | Platform | Introduced | How it works |
|---|---|---|---|
| Interleaved Reprojection | SteamVR | Before October 2016 | Forced the app to half frame rate (45 fps) and synthetically generated every second frame; Road to VR wrote that it had a negative impact on positional movement and animation; made obsolete by Motion Smoothing in 2018[8][25] |
| Asynchronous Reprojection | SteamVR | SteamVR Beta, 25 October 2016 | Valve's asynchronous reprojection, initially for NVIDIA GPUs only; Valve later presented Motion Smoothing as an improvement on it[25][6] |
| 60 Hz to 120 Hz reprojection | PlayStation VR | Described by Sony in 2015 | One of three rendering modes Sony offered developers by September 2015, alongside native 90 Hz and 120 Hz; Road to VR described it as "an interpolation method called 'asynchronous reprojection'"[19] |
| Asynchronous Spacewarp (ASW) | Oculus Rift (PC) | 10 November 2016 (runtime 1.10) | Extrapolates frames from previous frames; activated only when the app cannot hold the display rate; frames rendered 1/45 s apart produce a frame 1/90 s later[7] |
| Motion Smoothing | SteamVR | Beta 17 October 2018; release 27 November 2018 | Looks at the last two delivered frames, estimates motion and animation, and extrapolates a new frame; the app renders 1 of every 2 frames at 90 Hz on HTC Vive and HTC Vive Pro[6] |
| Motion reprojection | Windows Mixed Reality for SteamVR | Experimental (no date given by Microsoft) | Apps render at half rate (45 fps instead of 90) while "motion vectors generated by the GPU" are used to extrapolate the next frame[17] |
| Synchronous Spacewarp (SSW) | Virtual Desktop on Meta Quest 2 | June 2021 | Extrapolation runs on the headset rather than the PC, so it works with any GPU[21] |
| Application SpaceWarp (AppSW) | Quest headsets | November 2021 | App renders at half rate (for example 36 instead of 72 fps) and supplies motion vector and depth buffers used to synthesize frames[16] |
Valve designed Motion Smoothing to scale further when needed, synthesizing two frames for every frame the application delivers.[6] At launch, Motion Smoothing was not enabled with Oculus Rift or Windows Mixed Reality headsets in SteamVR, because Valve said "their underlying display drivers use different techniques when applications miss framerate".[6] Oculus said ASW "doesn't scale well below half the display's refresh rate" and that apps running at 90 fps on recommended hardware could typically run at 45 fps on minimum-spec systems with ASW covering the gap.[7] Microsoft has since deprecated Windows Mixed Reality, including Windows Mixed Reality for SteamVR, and removed it in Windows 11 version 24H2; existing headsets keep working with Steam through November 2026 on Windows 11 version 23H2.[17] In UploadVR's testing, Virtual Desktop's SSW gave "noticeably better extrapolation" than ASW; developer Guy Godin worked with Qualcomm on it, and it was not available on the original Quest.[21]
Application SpaceWarp moves part of the work to the application, which supplies ground-truth motion vectors. Meta reported that in its initial testing AppSW "gave applications up to 70 percent additional compute, potentially with little to no perceptible artifacts". Meta described the method as "frame extrapolation and frame reprojection, with the help of motion vector data and depth data". It supported refresh rates of 72, 90 and 120 Hz on Quest and Quest 2, with matching half-rate render targets of 36, 45 and 60 fps, and Meta provided a Unity reference implementation, an Unreal Engine 4.27 integration and an extension published in the OpenXR specification for native applications.[16]
Research
Video frame interpolation methods are compared on public datasets such as Middlebury, Vimeo-90K, UCF101 and the Xiph large-motion benchmark.[13][15] Selected work on motion estimation, interpolation and VR frame synthesis:
| Work | Venue | Contribution |
|---|---|---|
| Thomas, BBC RD 1987/11 | BBC Research Department report | Phase-correlation motion measurement for temporal standards conversion[2] |
| De Haan et al. | IEEE TCSVT, 1993 | 3-D recursive search block matching with eight candidate vectors per block[12] |
| Mark, McMillan, Bishop | I3D 1997 | Post-rendering 3D warping from color and depth to raise apparent frame rate and compensate for latency[18] |
| Choi et al. | IEEE TCSVT, 2007 | Motion-compensated frame interpolation with bilateral motion estimation[10] |
| Baker et al. | IJCV, 2011 | Optical flow benchmark that includes frame interpolation error[13] |
| Van Waveren | ACM VRST 2016 | Implementation trade-offs of asynchronous time warp on consumer hardware[11] |
| Niklaus, Mai, Liu | ICCV 2017 | Adaptive separable convolution kernels instead of explicit flow[1] |
| Jiang et al. (Super SloMo) | CVPR 2018 | Multi-frame interpolation with visibility maps for occlusion[9] |
| Guo et al. (ExtraNet) | ACM Transactions on Graphics, 2021 | Neural frame extrapolation to avoid interpolation latency[5] |
| Huang et al. (RIFE) | ECCV 2022 | Real-time intermediate flow estimation[14] |
| Reda et al. (FILM) | ECCV 2022 | Single network for large motion trained from frames alone[15] |
See also
References
- ↑ 1.0 1.1 1.2 1.3 Simon Niklaus, Long Mai, Feng Liu (2017). "Video Frame Interpolation via Adaptive Separable Convolution". 2017 IEEE International Conference on Computer Vision (ICCV), pp. 261-270. doi:10.1109/ICCV.2017.37. https://arxiv.org/abs/1708.01692. Retrieved 2026-10-06.
- ↑ 2.0 2.1 2.2 2.3 G. A. Thomas (1987-09). "Television motion measurement for DATV and other applications (BBC RD 1987/11)". BBC Research Department Report. British Broadcasting Corporation. https://downloads.bbc.co.uk/rd/pubs/reports/1987-11.pdf. Retrieved 2026-10-06.
- ↑ 3.0 3.1 3.2 3.3 3.4 Todd Spangler (2018-12-05). "Tom Cruise Explains Why HDTV 'Motion Smoothing' Is Terrible, Urges Movie Fans to Disable It". Variety. https://variety.com/2018/digital/news/tom-cruise-hdtv-motion-smoothing-disable-psa-1203080805/. Retrieved 2026-10-06.
- ↑ 4.0 4.1 4.2 4.3 4.4 Henry Lin, Andrew Burnes (2022-09-20). "NVIDIA DLSS 3: AI-Powered Performance Multiplier Boosts Frame Rates By Up To 4X". NVIDIA GeForce News. NVIDIA. https://www.nvidia.com/en-us/geforce/news/dlss3-ai-powered-neural-graphics-innovations/. Retrieved 2026-10-06.
- ↑ 5.0 5.1 5.2 Jie Guo, Xihao Fu, Liqiang Lin, Hengjun Ma, Yanwen Guo, Shiqiu Liu, Ling-Qi Yan (2021-12). "ExtraNet: Real-time Extrapolated Rendering for Low-latency Temporal Supersampling". ACM Transactions on Graphics, vol. 40, no. 6. https://doi.org/10.1145/3478513.3480531. Retrieved 2026-10-06.
- ↑ 6.0 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 Alex Vlachos (2018-11-27). "Introducing SteamVR Motion Smoothing". Steam News. Valve. https://steamcommunity.com/games/250820/announcements/detail/1705071932992003492. Retrieved 2026-10-06.
- ↑ 7.0 7.1 7.2 7.3 7.4 7.5 7.6 Dean Beeler, Ed Hutchins, Paul Pedriana (2016-11-10). "Asynchronous Spacewarp". Meta for Developers Blog. Meta. https://developers.meta.com/horizon/blog/asynchronous-spacewarp/. Retrieved 2026-10-06.
- ↑ 8.0 8.1 8.2 8.3 David Heaney (2019-01-17). "VR Timewarp, Spacewarp, Reprojection, And Motion Smoothing Explained". UploadVR. https://www.uploadvr.com/reprojection-explained/. Retrieved 2026-10-06.
- ↑ 9.0 9.1 9.2 9.3 9.4 Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz (2018). "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation". 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9000-9008. doi:10.1109/CVPR.2018.00938. https://arxiv.org/abs/1712.00080. Retrieved 2026-10-06.
- ↑ 10.0 10.1 Byeong-Doo Choi, Jong-Woo Han, Chang-Su Kim, Sung-Jea Ko (2007-04). "Motion-Compensated Frame Interpolation Using Bilateral Motion Estimation and Adaptive Overlapped Block Motion Compensation". IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, no. 4, pp. 407-416. https://doi.org/10.1109/TCSVT.2007.893835. Retrieved 2026-10-06.
- ↑ 11.0 11.1 11.2 J. M. P. van Waveren (2016-11). "The Asynchronous Time Warp for Virtual Reality on Consumer Hardware". Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (VRST '16), pp. 37-46. https://doi.org/10.1145/2993369.2993375. Retrieved 2026-10-06.
- ↑ 12.0 12.1 12.2 Gerard de Haan, Paul W. A. C. Biezen, Henk Huijgen, Olukayode A. Ojo (1993). "True-motion estimation with 3-D recursive search block matching". IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, no. 5, pp. 368-379. doi:10.1109/76.246088. https://research.tue.nl/en/publications/true-motion-estimation-with-3-d-recursive-search-block-matching/. Retrieved 2026-10-06.
- ↑ 13.0 13.1 13.2 Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, Richard Szeliski (2011-03). "A Database and Evaluation Methodology for Optical Flow". International Journal of Computer Vision, vol. 92, no. 1, pp. 1-31. https://doi.org/10.1007/s11263-010-0390-2. Retrieved 2026-10-06.
- ↑ 14.0 14.1 14.2 Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, Shuchang Zhou (2022). "Real-Time Intermediate Flow Estimation for Video Frame Interpolation". European Conference on Computer Vision (ECCV 2022). https://arxiv.org/abs/2011.06294. Retrieved 2026-10-06.
- ↑ 15.0 15.1 15.2 15.3 Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, Brian Curless (2022). "FILM: Frame Interpolation for Large Motion". European Conference on Computer Vision (ECCV 2022). https://arxiv.org/abs/2202.04901. Retrieved 2026-10-06.
- ↑ 16.0 16.1 16.2 16.3 Jian Zhang, Neel Bedekar, Leonard Tsai, Xiang Wei (2021-11-11). "Introducing Application SpaceWarp". Meta for Developers Blog. Meta. https://developers.meta.com/horizon/blog/introducing-application-spacewarp/. Retrieved 2026-10-06.
- ↑ 17.0 17.1 17.2 "Using SteamVR with Windows Mixed Reality - Enthusiast Guide". Microsoft Learn. Microsoft. https://learn.microsoft.com/en-us/previous-versions/mixed-reality/enthusiast-guide/using-steamvr-with-windows-mixed-reality. Retrieved 2026-10-06.
- ↑ 18.0 18.1 18.2 William R. Mark, Leonard McMillan, Gary Bishop (1997-04). "Post-Rendering 3D Warping". Proceedings of the 1997 Symposium on Interactive 3D Graphics (I3D '97), pp. 7-16. ACM. https://doi.org/10.1145/253284.253292. Retrieved 2026-10-06.
- ↑ 19.0 19.1 Ben Lang (2015-09-17). "Sony Confirms New 90Hz Display Mode for PlayStation VR (formerly Morpheus)". Road to VR. https://www.roadtovr.com/sony-confirms-new-90hz-display-mode-for-playstation-vr-formerly-morpheus/. Retrieved 2026-10-06.
- ↑ 20.0 20.1 20.2 Isaac Feldberg (2019-08-28). "Hollywood Heavies Unite Against TV 'Motion-Smoothing' With 'Filmmaker Mode' Setting". Fortune. https://fortune.com/2019/08/28/hollywood-directors-filmmaker-mode-motion-smoothing/. Retrieved 2026-10-06.
- ↑ 21.0 21.1 21.2 David Heaney (2021-06-10). "Virtual Desktop VR Streaming Gets 'Synchronous Spacewarp' On Quest 2". UploadVR. https://www.uploadvr.com/virtual-desktop-synchronous-spacewarp/. Retrieved 2026-10-06.
- ↑ 22.0 22.1 "AMD FSR 3 game integrations out now + more details for developers". AMD GPUOpen. AMD. 2023-09-29. https://gpuopen.com/news/fsr3-in-games-technical-details/. Retrieved 2026-10-06.
- ↑ "Filmmaker Mode FAQ". Filmmaker Mode. UHD Alliance. https://www.filmmakermode.com/faq/. Retrieved 2026-10-06.
- ↑ "AMD FidelityFX Super Resolution 3 (FSR 3)". AMD GPUOpen. AMD. https://gpuopen.com/fidelityfx-super-resolution-3/. Retrieved 2026-10-06.
- ↑ 25.0 25.1 "SteamVR Update Adds Asynchronous Reprojection". Road to VR. 2016-11-16. https://www.roadtovr.com/steamvr-update-adds-asynchronous-reprojection/. Retrieved 2026-10-06.