Cultural Heritage A hybrid human–computer approach for recovering incomplete cultural heritage pieces Antonio Ada ´n a , Santiago Salamanca b,n , Pilar Mercha ´n b a Escuela Superior de Informa ´tica, Universidad de Castilla la Mancha, Paseo de la Universidad, 4, Ciudad Real, Spain b Escuela de Ingenierı ´as Industriales, Universidad de Extremadura, Avda. de Elvas s/n, Badajoz, Spain article info Article history: Received 27 April 2011 Received in revised form 2 September 2011 Accepted 17 October 2011 Available online 3 November 2011 Keywords: 3D modeling Reconstruction Artificial intelligence Cultural heritage Human–Computer Interaction abstract The automatic reconstruction of archeological pieces through the integration of a set of unknown segments is a highly complex problem which is still being researched. When only a few segments of the original piece are available, solutions exclusively based on computational algorithms are inefficient when attempting to create a credible whole restoration. Incomplete 3D puzzles must consequently be tackled by considering hybrid human/computer strategies. This paper presents a reconstruction approach in which the knowledge of human experts and computational solutions coexist together. Hypotheses, models and integration solutions originating from both humans and computers are thus continuously updated until an agreement is reached. This semi-automatic restoration approach has been tested on a set of ancient fractured pieces belonging to the remains of Roman sculptures at the well known Me ´ rida site (Spain), and promising results have been obtained. The successful results and applicability of this method have led us to believe that computational solutions should evolve towards hybrid human–computer strategies. & 2011 Elsevier Ltd. All rights reserved. 1. Solutions to 3D puzzles: state of the art The automatic reconstruction of original pieces from their fragments is an extremely common problem in sciences such as archeology, paleontology, art restoration, etc. Attempts have formerly been made to manually reconstruct ancient pieces by considering a set of characteristics, principally based on geometry and color. However, this becomes a tedious and difficult task when many segments have to be assembled in the right position. In a 2D environment, as is the case of mosaics, humans have a certain amount of skill in discovering coherent solutions which will eventually allow them to reconstruct middle-difficulty jig- saws, although this is highly time-consuming. Nevertheless, assembling fragments for 6 Degree Of Freedom (DOF) cases is an extremely difficult problem for humans. For this very reason, it is crucial to find methods that will allow these tasks to be performed with the aid of artificial intelligence algorithms. A computer should therefore select the fragments which belong to the query piece and calculate their correct pose in it. The entire object can be then reconstructed. This objective can be accomplished by extracting precise 3D models of the fragments and using certain characteristics such as color, texture, shape or dimensions, among others. Three principal solid reconstruction techniques – classified by the kind of archeological environment – can be found in journals and conferences from the last two decades: 1. Works and projects in which the fragments belong to original flat shapes, such as roads, walls, floors and mosaics. In this case, the problem of assembling these 3D shape fragments can be solved with 2D jigsaw techniques. For example, Leitao and Stolfi [1] have devised a method for the automatic reassembly of 2D fragments. In this method, the boundaries of the fragments are represented as curvature codes. The fragments are assembled by comparing these curvature codes in order to find real matching candidates. This is useful in the reconstruc- tion of aspects of archeological sites such as tiles and murals. The results of this research are satisfactory, with a greater degree of accuracy in some materials than others, but the method can only be used to perform reconstructions of flat- shaped objects. Hori et al. [2] propose a method based on the partial verification of the similarities between two pairs of contours. They also apply the method to 2D images rather than 3D data. A similar method is presented by Kanoh et al. in [3]. In the first step, the authors join the potsherds in two dimen- sions. The contours are divided into sub-contours by salient points. These sub-contours are described by P-Type Fourier descriptors [4]. The matching is carried out by comparing Contents lists available at SciVerse ScienceDirect journal homepage: www.elsevier.com/locate/cag Computers & Graphics 0097-8493/$ - see front matter & 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.cag.2011.10.003 n Corresponding author. Tel.: þ34 924289300; fax: þ34 924289601. E-mail addresses: Antonio.Adan@uclm.es (A. Ada ´ n), ssalaman@unex.es (S. Salamanca), pmerchan@unex.es (P. Mercha ´ n). Computers & Graphics 36 (2012) 1–15