Reconstruction of a Three-Dimensional Tableau from a Single Realist Painting Wai L. Khoo * , Tadeusz Jordan † , David G. Stork ‡ and Zhigang Zhu § *†§ Department of Computer Science, CUNY City College Convent Avenue and 138th Street, New York NY 10031 Email: * wlkhoo@gmail.com. † tedjj123@gmail.com, § zhu@cs.ccny.cuny.edu ‡ Ricoh Innovations, 2882 Sand Hill Road, Suite 115, Menlo Park CA 94025-7054 Department of Statistics, Stanford University, Stanford CA 94305 Email: artanalyst@gmail.com Abstract—We reconstructed a three-dimensional tableau from a single realist painting–Scott Fraser’s “Three way vanitas” (2006)–based on multiple stereo reconstruction applied to the direct image and the images in three plane mirrors depicted within the painting. The tableau contains a carefully chosen complex arrangement of objects including a moth, egg cup, and strand of string, glass of water, bone, and hand mirror. Each of the three plane mirrors presents a different view of the tableau from a virtual camera behind each mirror and symmetric to the artist’s viewing point. Our new contributions are three folds. First, we incorporate single-view geometric information extracted from the direct image of the wooden mirror frames in obtaining the camera models of these three virtual cameras. Second, we estimate 3D of objects using multiple stereo pairs within a single painting. Third, the geometric accuracies of the painting are also evaluated. I. I NTRODUCTION The problem of reconstructing a three-dimensional scene from multiple views is well explored, and a number of general methods, such as those based on correlation, relaxation, dy- namic programming, have been developed and fully character- ized [1], [2]. Three-dimensional reconstruction and metrology can be based on single views as well [3], [4], and recently Criminisi and his colleagues have applied such techniques to the analysis of paintings, for instance reconstructing the virtual spaces in Masaccio’s Holy Trinity (c. 1425), Piero della Francesca’s Flagellation of Christ (c. 1453), Hendrick V. Steenwick’s St. Jerome in his study (1630), Jan Vermeer’s A lady at the virginals with a gentleman (16621665), and others [5]. These methods reveal both the high geometric accuracies in some passages, and the geometric inconsistencies in others, properties that are nearly impossible to determine by eye. Such analyses shed new light on these works and the artists’ working methods, for instance revealing whether an artist likely used geometrical aids during the execution of their work. Recently Smith, Stork and Zhang reconstructed the three- dimensional space depicted in a highly realistic modern paint- ing, Scott Fraser’s Three way vanitas (Fig. 1) using traditional multiple-view reconstruction methods applied to the direct view and a view visible in a depicted mirror [6]. They found passages within the painting having good spatial agreement, showing high accuracy of the painter, but also passages of significant disagreement, such as the height of the water in the glass in the direct view and the view in the right-hand mirror. Even though using reflected images by mirrors is a very popular approach for stereo vision in computer vision [7]–[10], it was the first time to analyze a painting with such a setup. There were some limitations in that previous work as well as unexplored opportunities. For instance, each reconstruction was based on just the direct view and a single reflected view. In fact, though, several volumes within the tableau are visible in three or more views and could have been reconstructed using all those views. Furthermore, the images of the frames of the mirrors provide geometric constraints about the centers of projection of the images depicted within each mirror, and the earlier scholarship did not incorporate that information when reconstructing the three-dimensional space. It is this latter opportunity we address here. However, 3D reconstruction is not a goal in itself, but merely to further evaluate the painting accuracy estimation. The paper is organized as the following. Section II describes the painting, previous scholarship and an overview of our new approach to three-dimensional reconstruction based on incorporating single-view geometrical constraints into multi- view image correspondences. Section III briefly describe the single-view geometric estimation of camera parameters. Sec- tion IV verifies the accuracy of the painting based on multi- view epipolar geometry. Section V further verifies the accu- racy of the painting based on multi-view three-dimensional reconstruction. Section VI summarizes our conclusions and describes future directions. II. THE WORK AND PROBLEM ADDRESSED Fig. 1 shows the work we consider, Scott Fraser’s Three way vanitas (2006). This painting was commissioned as part of The Object Project [11], in which fifteen artists were commissioned to create works, each containing five specified objects: hand