header for SPIE use Registration and Integration of Multi-Sensor Data for Photo-realistic Scene Reconstruction Faysal Boughorbal *a,b , David L. Page b , Christophe Dumont b,c , Mongi A. Abidi b a Ecole Nationale des Ingenieurs de Tunis, Tunisia b Imaging, Robotics, and Intelligent Systems Labortory, University of Tennessee, Electrical Engineering, Ferris Hall, Knoxville, Tennessee, 37996-2100, USA c Université de Bourgogne, 12 rue de la fonderie, 71200 LeCreusot, France ABSTRACT In this paper, we present a method for automatically registering a 3D range image and a 2D color image using the χ 2 - similarity metric. The goal of this registration is to allow the reconstruction of a scene using multi-sensor information. Traditional registration algorithms use invariant image features to drive the registration process. This approach limits the applicability to multi-modal data since features of interest may not appear in each modality. However, the χ 2 -similarity metric is an intensity-based approach that has interesting multi-modal characteristics. We explore this metric as a mechanism to govern the registration search. Using range data from a Perceptron laser camera and color data from a Kodak digital camera, we present results using this automatic registration with the χ 2 -similarity metric. Keywords: multi-sensor registration, range and color registration, 3D scene reconstruction 1. INTRODUCTION The reconstruction of 3D multi-modal indoor scenes requires the use of both geometric and intensity data. Often, the geometry data are in the form of 3D (or 2½D) range maps while the intensity data are usually 2D images. In the case of photo-realistic reconstruction, color photographs serve as the 2D intensity image. Other applications may require additional modalities besides color (thermal, radiometric, etc.). Data fusion 1 of these modalities and the geometry information leads to a multi-modal scene representation. This representation is interesting for object recognition and other vision tasks because it provides more information about objects in the scene than a single geometric reconstruction or 2D image, alone. An important step in data fusion is registration, or alignment, of the sensor coordinate systems. In this work, our goal is to use color images (photographs) as texture maps for a geometric model built from range images. Since we are using unregistered data sets, a necessary step in this process is to align the two sensors (laser range camera and color camera). This alignment problem requires the estimation of the pose of one sensor relative to the other. To facilitate on-line processing, an automatic registration algorithm is needed. The experiment presented in this paper involves the registration of data from a Perceptron P5000 laser range camera and a Kodak DCS460 digital color camera (see Figure 1). The range image allows for the construction of a 3D surface model of the scene (see Figure 2). This model is basically a triangle mesh that serves as an anchor for the color texture map from the digital camera. The range image exhibits important dissimilarities with the color image. The relationship between the brightness values of corresponding pixels is complex. However, the Perceptron outputs an intensity image along with the range image. This image represents the relative strength of the returning laser after striking the surface of an object. This intensity image is what we use to relate to the color image (see Figure 3). Also, aligning the 3D to 2D data presents additional difficulties due to the occlusion problem. The objective of this experiment is to overcome these problems and to use the range data and the color data to find the relative pose of the Perceptron to the Kodak camera. * Correspondence: Email: boughor@iristown.engr.utk.edu; WWW: http://iristown.engr.utk.edu , Telephone: 423 974 9213, Fax: 423 974 5459 F. Boughorbel, D. Page, C. Dumont, and M. A. Abidi, "Registration and Integration of Multi-Sensor Data for Photo-realistic Scene Reconstruction" Proc. of SPIE Conf. on Applied Imagery Pattern Recognition, Vol. 3905, pp. 74-84, Washington, D.C., October 1999. 96 74 In 28th AIPR Workshop: 3D Visualization for Data Exploration and Decision Making, William R. Oliver, Proceedings of SPIE Vol. 3905 (2000) 0277-786X/00/$15.00