A FRAMEWORK FOR THE REGISTRATION OF COLOR IMAGES WITH 3D MODELS Louis Borgeat, Guillaume Poirier, Angelo Beraldin, Guy Godin, Philippe Massicotte and Michel Picard National Research Council of Canada ABSTRACT This paper describes an environment to automatically or semi-automatically compute the precise mapping between a set of 2D images and a triangulated 3D model built from high- resolution 3D range data. This environment is part of our Ate- lier3D framework for the modeling, visualization and analysis of large sensor-based datasets. This work was done to initially support three cultural heritage application projects: the mod- eling of the Grotta dei Cervi in Italy, of the Erechtheion in Athens, Greece, and of Leonardo’s Mona Lisa. The proposed method combines image-based registration, feature matching, robust estimation techniques and advanced multi-resolution rendering with a powerful user interface. Index Terms— Image registration, cultural heritage, large datasets, 3D sensors, texture mapping 1. INTRODUCTION As sensor technology improves, it is now feasible to acquire vast amounts of 2D and 3D data from cultural sites in a very short time. Modern sensors can acquire billions of 3D sam- ples and terabytes of pixel data in a matter of hours. However, the process of transforming all the raw data into one accurate model can still be very time and resource consuming and can quickly become a major project bottleneck. In this paper we address the issue of having to precisely map a large number of high-resolution digital photographs onto a 3D model in order to produce a seamless integrated color model for visualisation and analysis. The described processing is part of the steps that prepare the data for visualization within the Atelier3D frame- work [1]. This frameworks aims at providing a complete set of tools for the efficient and acurate modeling, visualisation and analysis of large sensor-based 3D datasets. Atelier3D al- ready permits visualisation and analysis of large models on desktop hardware. We are now in the process of integrating modeling components within this environement. Numerous techniques have been proposed to address the problem of automatic registration between 3D and 2D images. Hantak[2] provides a good litterature review and classification of automatic registration techniques and com- pares different image-based similarity metrics for 3D/2D Special thanks to Prof. V. Valzano and A. Bandiera, Coordinamento SIBA, Universit del Salento, Lecce, Italia. intensity registration. Typically, proposed methods itera- tively use image-based registration followed by a new pose estimation[3][4] until convergence. Variations involve fea- ture or edge detection in the images and direct intensity comparison using different metrics and optimizers. Little work presents results for larger image databases, which are in practice still often processed using highly interactive soft- ware relying on manual selection of feature points between images. What we propose in this paper is a set of tools that enable an application user to process rapidly large amounts of tex- ture data for a broad range of application contexts. Indeed, the problem of 2D/3D registration can vary significantly be- tween types of practical applications. In some cases the in- tensity image obtained from the 3D data will be very similar to the corresponding 2D image and in other cases quite differ- ent and therefore harder to register; many older range sensor do not even provide such intensity information. If the 2D and 3D sensors are physically attached, we get a good ini- tial pose and camera calibration, but in other cases one must work only from randomly taken photographs. No single al- gorithmic solution can currently cover all acquisition setups and physical environments. What we propose is a framework to tackle most situations with minimal user intervention. We will first describe an interactive graphical user environment to rapidly produce a good initial alignment/calibration and to validate results. We then describe a set of automatic tools to iteratively refine an initial estimate until sufficient accuracy is achieved. We finally present results for a few application projects. Here, our main original contribution lies more in the proposed system for large dataset processing than in a new specific individual algorithm. As such, we have tested it for different practical applications in the heritage field character- ized by 3D datasets composed of hundreds of millions of tri- angles and large texture databases. 2. IMAGE REGISTRATION The interactive registration process that leads to a first pose and camera calibration goes as follows (some steps can be skipped or repeated depending on the quality of the initial alignment): First, the user aligns a semi-transparent version of the 3D model with the image using mouse-based navigation. The 69 978-1-4244-5654-3/09/$26.00 ©2009 IEEE ICIP 2009