To appear in the 6th IEEE International Conference on Automatic Face and Gesture Recognition. May 2004. Multi-biometrics Using Facial Appearance, Shape and Temperature Kyong I. Chang Kevin W. Bowyer Patrick J. Flynn Xin Chen Computer Science & Engineering Department University of Notre Dame Notre Dame, IN 46556 U.S.A. {kchang,kwb,flynn}@cse.nd.edu Abstract We present results of the first study to examine individual and multi-modal face recognition using 2D, 3D and infra- red images of the same set of subjects. Each sensor cap- tures different aspects of human facial features; appearance in intensity representing surface reflectance from a light source, shape data representing depth values from the cam- era, and the pattern of heat emitted, respectively. We em- ploy a database containing a gallery set of 127 images and an accumulated time-lapse probe set of 297 images. Using a PCA-based approach tuned separately for 2D, 3D and IR, we find rank-one recognition rates of 90.6% for 2D, 91.9% for 3D and 71.0% for IR. Combining each pair of modal- ities, we find a multi-modal rank-one recognition rate of 98.7% for 2D-3D, 96.6% for 2D-IR and 98.0% for 3D-IR. When all three modalities are combined, we obtain 100% recognition. The results shown in this study appear to sup- port the conclusion that the path to higher accuracy and robustness in biometrics involves use of multiple biometrics rather than the best possible sensor and algorithm for a sin- gle biometric. 1. Introduction Face is one of the most important and commonly used bio- metrics for identification due to its acceptability, universal- ity and non-intrusiveness [1]. The identification of the hu- man face in 2D has been investigated by many researchers, but relatively few studies using other aspects of facial fea- tures have been reported. Each imaging modality has its own benefits and prob- lems when applied to face recognition. 2D images are gen- erally easier and less expensive to acquire. The perceived benefits from using 3D relative to 2D data include less vari- ation observed due to factors such as makeup and reduced sensitivity to illumination changes (even though a 3D sens- ing operation is influenced by the illumination). Also, the pattern of heat emitted from the human body (face) may effectively be considered as a characteristic of each individ- ual [2]. This paper first addresses the effectiveness of each individual biometric source, using a PCA-based technique to investigate the identification accuracy of the approach. Then, we consider the combination of different facial fea- tures. This is the first study to examine these three face biometrics and to compare the different pairs of face bio- metrics. Even though each imaging modality has its own advantages and disadvantages depending on certain circum- stances, there is often some expectation that 3D data should yield better performance. However, no rigorous experimen- tal study has been reported to validate this expectation. The experiments reported in this study are aimed at (1) testing the hypothesis that there exists a superiority of accuracy for one biometric over other biometric sources, using the PCA- based method, and (2) exploring whether a combination of 2D, 3D and IR face data may provide better performance than any one individually. One aspect of combining dif- ferent biometrics is how to combine results provided by individual sources effectively during the decision process. Many different approaches could be envisioned for com- bining multiple types of biometric information [3, 4, 5]. In general, they can be thought of as occurring at the image level, the metric level, or the rank level. In this study, we consider multi-modal combination at the metric level. The term “multi-modal biometrics” is used here to refer to the use of different sensor types without necessarily indicating that different parts of the body are used. Important aspects of some related multi-modal studies are summarized in Ta- ble 1. 1