Image Inpainting Applied to Art Completing Escher’s Print Gallery Lucia Cipolina-Kun 1 Simone Caenazzo 2 Gaston Mazzei 31 Aditya Srinivas Menon 4 Abstract This extended abstract presents the first stages of a research in inpainting suited for art reconstruc- tion. We introduce M.C Escher’s Print Gallery lithography as a use-case example. This artwork presents a void on its center and additionally, it follows a challenging mathematical structure that needs to be preserved by the inpainting method. We present our work so far and our future line of research. 1. Introduction Completing missing sections of a painting poses several difficulties for the current state-of-the-art knowledge in AI. The model is required to learn how to paint like a certain artist, in a certain period, on a given painting and potentially continue it seamlessly as if the painter had a bigger canvas. We aim to develop a method that goes beyond learning the style of a given artist - see for example the Style Transfer models in Gatys et al. (Gatys et al., 2015) - and learn the idiosyncrasies of a single given artwork. This allow us to reconstruct the painting’s missing pieces following the original style, content and context of the piece. As an application to our model, we chose Escher’s Print Gallery lithography as it poses an interesting blending of art and mathematics, as well as the above mentioned challenges of an inpainting model. Firstly, any attempt to complete the painting will have to preserve its unique mathematical properties. Secondly, this painting features a very unique style, effectively found only in a handful of works by Es- cher; there are therefore not enough samples to perform a straightforward fine-tuning of typical pre-trained models, such as Image-GPT (Chen et al., 2020) or Taming Trans- formers (Esser et al., 2020) which require thousands of training images. 1 ML Collective 2 Riskcare Ltd., London 3 Departamento de Fisica, Universidad de Buenos Aires, Buenos Aires, Argentina 4 Indian Institute of Information Technology, Kottayam. Corre- spondence to: Lucia Cipolina-Kun <lcipolina@gmail.com>. Proceedings of the LatinX in AI (LXAI) Research workshop at ICML 2021. Copyright 2021 by the author(s). 2. Related Work The application of image completion and inpainting to fine art and painting reconstruction is still an incipient field, with very few examples to date like (Zeng et al., 2019) reconstructing Van Gogh’s degraded painting or (Gupta et al., 2019) for restoration in digital art. In these works, the aim is not to create new content to fill-in missing sections of an image, but to restore existing content, degraded sections or lines. 3. Escher’s Print Gallery The work Print Gallery (Dutch: Prentententoonstelling), shown in Figure 1, is a lithography made in 1956 by the Dutch artist M. C. Escher. It depicts a man looking at painting on a gallery placed in the Grand Harbour of Valletta (Malta). At the center of the lithography, Escher left a blank space with his signature. The reason for this blank area is un- known: some authors, e.g. (Hofstadter), claim that he was not actually able to fill that space, while (Ernst, 1976) claims that it is not even possible to continue the spiral-like effect in a sensible way. 3.1. The Mathematical Structure Behind Print Gallery Print Gallery poses an additional challenge for inpainting models as it has an underlying mathematical structure that needs to be preserved when continuing the lithography. As explained in (Ernst, 1976) and (de Smit & Lenstra Jr, 2003), the lithography features a particular sequence of roto- homotheties known in art jargon as a Droste effect: i.e., the image contains a repetition of itself with a spiral-like motion inducing the idea of an infinite loop. Additionally to this self-repeating structure, the lithography features a wave-like spatial distortion. To achieve this effect, Escher used a grid-like pattern based on curved lines, to enhance the idea of circular motion already embedded in the Droste effect. Figure 2 shows the grid structure used by Escher superimposed to the painting - additionally, in the center of the image, we show the corresponding Droste repetition as explained in (de Smit & Lenstra Jr, 2003).