Volume xx (200y), Number z, pp. 1–8 Patchmatch for Texture Synthesis Daniel Prieler, Stefan Jeschke (Advisor) Vienna University of Technology, Austria 1. Introduction and Related Literature This paper is a report about a project practical course (“Pro- jektpraktikum”) done at the Institute for Computer Graph- ics and Algorithms of the Vienna University of Technol- ogy. Advisor: Stefan Jeschke. The goal of this course was to implement a texture synthesis algorithm and analyze its performance and visual quality. Two papers form the theo- retical foundation of this project: “Texture Optimization for Example-based Synthesis” by Kwatra et al [KEBK05] and “Patchmatch: A randomized Correspondence Algorithm for Structural Image Editing” by Barnes et al [BSFG09]. 1.1. Texture Optimization for Example-based Synthesis The technique proposed in this paper [KEBK05] works on the whole image at once. It defines a metric for similar- ity between two images, which is called texture energy. The algorithm then tries to minimize it iteratively. In short, this energy compares local neighborhoods of the input tex- ture to patches in the synthesized image (the terms patches and neighborhoods are used interchangeably). In contrast to region-growing methods this energy is a global metric for the whole texture. The texture energy is defined formally as follows: E t (x;  z p  )= ∑ p∈X † ‖x p − z p ‖ 2 (1) In this notion X is the set of all pixels of the synthesized texture and Z would be the example texture. The lower case x and z are the vectors of the pixels of X and Z respectively. x p and z p are vectors of length w, which contain some pixels of x and z. These correspond to little regions in an image. The index p indicates the pixel p which is the center of the current patch (or region). The similarity metric works as fol- lows: Take two patches - one of the synthesized and one of the input texture - and take the Euclidean norm of the two. (Basically the distance between the two vectors consisting of w pixels) Do this for all patches of the synthesized texture and sum up all the distances. It is not necessary to perform these steps for every pixel, (not every pixel is the center of a patch). Kwatra et al claimed that using only every w 4 pixel increases both visual quality and performance. (X † is the set of pixels of the output texture which are centers of those neighborhoods which are taken into account with the energy function) Image synthesis can be carried out by minimizing this texture energy. At first an initial guess of the output tex- ture has to be made; each neighborhood in the output texture is assigned randomly to a neighborhood of the sample tex- ture. After this initialization step the algorithm progresses iteratively: • The new output texture (x) is determined by taking all neighborhoods of the sample texture (z p ) (which are cur- rently chosen) and combine them so that the texture en- ergy is minimized. • The new neighborhoods of the sample texture are com- puted by finding the nearest neighbor of every patch in the output texture (x p ) in the sample texture. These steps are repeated until a fixed number of times or if the neighborhood of the sample texture does not change anymore. The idea of not taking every pixel of the output texture into account has two positive effects: On the one hand it reduces the number of computations, since the near- est neighbor search (which is quite expensive) is not used that often. On the other hand the quality of the synthesized texture is greater as well: By using fewer patches, each pixel is influenced by a smaller number of neighborhoods which produces better results when areas overlap that do not fit to each other. As a result these regions are less blurry. In order to improve the convergence of the optimization pro- cess one can adapt the energy metric 1 of the algorithm: The Euclidean norm performs similar to a least square ap- proximation of the synthesized texture. Fitting a line in an over-determined system is quite prone to outlier. Similarly a single non-fitting pixel can deteriorate the speed of con- vergence significantly. To address this problem a different distance metric can be used: Coleman et al [CHK ∗ 80] sug- gested a method called “Iteratively reweighted least squares” (IRLS) which is used by Kwatra et al. This adds a weight- submitted to COMPUTER GRAPHICS Forum (9/2011).