Fragment-Based Image Completion Iddo Drori Daniel Cohen-Or Hezy Yeshurun School of Computer Science ∗ Tel Aviv University Figure 1: From left to right: the input image, and inverse matte that defines the removal of an element, the result of our completion, and the content of the completed region. Abstract We present a new method for completing missing parts caused by the removal of foreground or background elements from an image. Our goal is to synthesize a complete, visually plausible and coher- ent image. The visible parts of the image serve as a training set to infer the unknown parts. Our method iteratively approximates the unknown regions and composites adaptive image fragments into the image. Values of an inverse matte are used to compute a confidence map and a level set that direct an incremental traversal within the unknown area from high to low confidence. In each step, guided by a fast smooth approximation, an image fragment is selected from the most similar and frequent examples. As the selected fragments are composited, their likelihood increases along with the mean con- fidence of the image, until reaching a complete image. We demon- strate our method by completion of photographs and paintings. CR Categories: I.3.3 [Computer Graphics]: Picture/image generation—; I.4.1,3,5 [Image Processing and Computer Vision]: Sampling—,Enhancement,Reconstruction Keywords: image completion, example-based synthesis, digital matting, compositing ∗ e-mail: {idrori | dcor | hezy}@tau.ac.il 1 Introduction The removal of portions of an image is an important tool in photo- editing and film post-production. The unknown regions can be filled in by various interactive procedures such as clone brush strokes, and compositing processes. Such interactive tools require meticulous work driven by a professional skilled artist to complete the image seamlessly. Inpainting techniques restore and fix small- scale flaws in an image, like scratches or stains [Hirani and Totsuka 1996; Bertalmio et al. 2000]. Texture synthesis techniques can be used to fill in regions with stationary or structured textures [Efros and Leung 1999; Wei and Levoy 2000; Efros and Freeman 2001]. Reconstruction methods can be used to fill in large-scale missing re- gions by interpolation. Traditionally, in the absence of prior knowl- edge, reconstruction techniques rely on certain smoothness assump- tions to estimate a function from samples. Completing large-scale regions with intermediate scale image fragments remains a chal- lenge. Visual perceptual completion is the ability of the visual system to “fill in” missing areas [Noe et al. 1998] (partially occluded, coin- ciding with the blind spot, or disrupted by retinal damage). While the exact mechanisms behind this phenomenon are still unknown, it is commonly accepted that they follow some Visual Gestalt [Koffka 1935, 1967] principles, namely, completion by frequently encoun- tered shapes that result in the simplest perceived figure [Palmer 1999]. Motivated by these general guidelines, we iteratively ap- proximate the missing regions using a simple smooth interpolation, and then add details according to the most frequent and similar ex- amples. Problem statement: Given an image and an inverse matte, our goal is to complete the unknown regions based on the known re- gions, as shown in the figure above. In this paper, we present an iterative process that interleaves smooth reconstruction with the synthesis of image fragments by example. The process iteratively generates smooth reconstructions to guide the completion process which is based on a training set derived from the given image context. The completion process consists of compositing image frag- Permission to make digital/hard copy of part of all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date appear, and notice is given that copying is by permission of ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. © 2003 ACM 0730-0301/03/0700-0303 $5.00 303