INVITED PAPER Face Recognition Using 2-D, 3-D, and Infrared: Is Multimodal Better Than Multisample? Use of multiple images of a face appears to improve recognition accuracy regardless of the type of images that are taken. By Kevin W. Bowyer, Fellow IEEE, Kyong I. Chang , Patrick J. Flynn, Senior Member IEEE, and Xin Chen ABSTRACT | This work examines face recognition using normal intensity images, infrared images, three-dimensional shape, and combinations of these. We compare the performance improvement obtained by combining three-dimensional or infrared with normal intensity images (a Bmultimodal[ ap- proach) to the performance improvement obtained by using multiple intensity images (a Bmultisample[ approach). Com- bining results from different types of imagery gives signifi- cantly higher recognition rates than are obtained by using a single intensity image. However, significantly higher recogni- tion rates are also obtained by combining results from multiple intensity images. Overall, initial results indicate that, using an Beigen-face[ recognition algorithm and weighted score fusion, multisample techniques can result in a performance increase comparable to that of multimodal techniques. KEYWORDS | Biometrics; face recognition; information fusion; infrared; multimodal; three-dimensional I. INTRODUCTION The vast majority of face recognition research assumes that an attempt to recognize a person is made using a single intensity image of the type taken by standard cameras. A recent broad survey of such face recognition research is given by Zhao [1]. However, evaluations such as the 2002 Face Recognition Vendor Test (FRVT) [2] have shown that the accuracy of face recognition is not yet sufficient for the more demanding applications. Compli- cations that arise from variations in pose, lighting, and facial expression are among the various factors that contribute to decreased performance. This has led some researchers to investigate the use of three-dimensional (3-D) shape information for face recognition [3]–[8]. Some motivations for using 3-D shape are that shape is defined independent of lighting, and that acquiring 3-D shape should allow for accurate pose correction. Other researchers have investigated the use of infrared (IR) images for face recognition [9]–[12]. A major motivation for using IR images is that they are relatively unaffected by changes in lighting. Examples of these different types of face image appear in Fig. 1. In addition to exploring the use of 3-D shape and IR images as alternatives to normal intensity images, researchers have developed approaches to combining the recognition results from either a 3-D shape model or an IR image with the results from an intensity image. These approaches that combine different types of information for face recognition are commonly, although perhaps imprecisely, referred to as multimodal. A general meaning of the term multimodal is simply that different properties are sensed. A more specific possible meaning is that dif- ferent sensors are used in acquiring the data. Some sensors that give integrated 3-D shape and intensity image Manuscript received September 29, 2005; revised June 29, 2006. This work was supported in part by the National Science Foundation under Grant CNS-0130839, in part by the Central Intelligence Agency, in part by the National Geo-Spatial Intelligence Agency, in part by UNISYS Corp., and in part by the U.S. Department of Justice under Grant 2005-DD-BX-1224 and Grant 2005-DD-CX-K078. K. W. Bowyer and P. J. Flynn are with the Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556 USA (e-mail: kwb@cse.nd.edu; flynn@cse.nd.edu). K. I. Chang is with the Philips Medical SystemsVUltrasound, Bothell, WA 98041 USA (e-mail: jin.chang@philips.com). X. Chen is with the Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556 USA and also with Navteq, Chicago, IL 60654 USA (e-mail: xchen2@cse.nd.edu; xin.chen@navteq.com). Digital Object Identifier: 10.1109/JPROC.2006.885134 2000 Proceedings of the IEEE | Vol. 94, No. 11, November 2006 0018-9219/$20.00 Ó2006 IEEE