EUROGRAPHICS 2012 / P. Cignoni, T. Ertl (Guest Editors) Volume 31 (2012), Number 2 Iterative Image Warping Huw Bowles 1 Kenny Mitchell 2 Robert W. Sumner 2 Jeremy Moore 1 Markus Gross 2,3 1 Disney Interactive Studios 2 Disney Research Zurich 3 ETH Zurich Abstract Animated image sequences often exhibit a large amount of inter-frame coherence which standard rendering algo- rithms and pipelines are ill-equipped to exploit, limiting their efficiency. To address this inefficiency we transfer rendering results across frames using a novel image warping algorithm based on fixed point iteration. We ana- lyze the behavior of the iteration and describe two alternative algorithms designed to suit different performance requirements. Further, to demonstrate the versatility of our approach we apply it to a number of spatio-temporal rendering problems including 30-to-60Hz frame upsampling, stereoscopic 3D conversion, defocus and motion blur. Finally we compare our approach against existing image warping methods and demonstrate a significant performance improvement. Categories and Subject Descriptors (according to ACM CCS): I.3.m [Computer Graphics]: Miscellaneous—Image- based rendering 1. Introduction In computer graphics significant coherence is exhibited across frames of an animation (temporal coherence) and across nearby views of a scene (spatial coherence). Current rendering pipelines recompute each frame from scratch, re- sulting in a large amount of repeated work in processing the scene geometry and in performing the light simulation necessary to shade the surfaces. This inefficiency imposes severe constraints on the visual fidelity of real-time appli- cations in which rendering budgets are measured in mere milliseconds. Consequently enormous time and effort are in- vested in optimization of algorithms and assets which in- creases rendering pipeline complexity, complicates content creation workflows and drives up production costs. Although previous work includes a number of approaches to re-use information across frames to exploit this redun- dancy, they are yet to meet the rigorous requirements of production scenarios and have therefore eluded widespread adoption. These methods incur a heavy geometry processing cost or require significant changes to rendering pipelines and workflows. In this work we argue for the reuse of shading information across frames through image warping techniques. To trans- fer this information efficiently we introduce a novel image warping algorithm based on fixed point iteration. We de- scribe a heuristics-based variant of our approach that runs with minimal performance overhead. Additionally from the mathematical properties of fixed point iteration we derive a local convergence criterion which informs a robust adaptive variant of our approach which requires a little more com- putation time but still a fraction of the time of comparable methods. Our approach can be attached to existing render- ing pipelines with minimal side effects, which we demon- strate by installing our warping kernel into an existing AAA console game to post convert it stereoscopic 3D and to up- sample its frame rate from 30 to 60Hz. Finally we compare our method to existing work and demonstrate a substantial performance improvement without loss of quality. 2. Background In this section we review the available options for exploit- ing frame to frame coherence. Reverse reprojection caching (RRC) [NSL ∗ 07, SaLY ∗ 08] rasterizes the scene from the tar- get viewpoint and computes at each pixel the inverse map- ping from target to source image frame. Although the au- thors report reduced fragment shader cost, the scene geom- etry needs to be reprocessed once or twice per reprojec- tion which is prohibitively expensive for many production scenes. Additionally, shader authoring is complicated by the need to write variants of the vertex and fragment programs of each shader instrumented with the required reprojection code. c  2012 The Author(s) Computer Graphics Forum c  2012 The Eurographics Association and Blackwell Publish- ing Ltd. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA. DOI: 10.1111/j.1467-8659.2012.03002.x