Color Effect on the Face Recognition with Spatial Resolution Constraints Jae Young Choi 1,2 , Seungji Yang 1 , Yong Man Ro 1,2 , Konstantinos N. Plataniotis 2 1 Image and Video System Laboratory, Information and Communication University, 2 Department of Electrical and Computer Engineering, University of Toronto vanchoi@icu.ac.kr Abstract In the practical face recognition (FR) applications, low-resolution faces (20  20 pixels or less) are commonly encountered and negatively impact on reliable performance. To overcome low-resolution face problem, we show that face color can significantly improve the performance compared to intensity-based features. The contribution of this paper is twofold. First, a new metric called ‘variation ratio gain’ (VRG) is proposed to theoretically prove the significance of color effect on low-resolution faces. Second, we conduct extensive performance comparison studies. In particular, 3,192 color facial images corresponding to 341 subjects, collected from three standard CMU PIE, FERET, and XM2VTSDB face databases, were used to perform comparative studies of color effect on various face resolutions. Experimental results verified that face color feature improves the degraded recognition rate due to low-resolution faces by at least an order of magnitude over intensity-based features. 1. Introduction Face recognition technologies are being revisited toward Multimedia Information Retrieval (MIR) [1]. With increasing demands of automatic annotation of faces for personal photos, snap-shot images and video clips offered via Web services, FR technologies have been central part for reliable annotation of faces on various multimedia contents. Despite recent growth, precise FR is still a tough task due to faces captured from various environments including illumination, pose, aging, and resolution variations. In particular, many current FR-based multimedia applications often suffer from small-sized faces ( 20 20 × pixels or less) [2-4] from limited capturing conditions, e.g., faces captured from long distance cameras or camera-phones. Some FR literatures dealt with face resolution problem [2-3], [5-7]. In [2], 15 15 × pixels is considered to be as a minimum face resolution for reliable detection and recognition of faces. The CHIL project [5] reported that normal face resolution in video-based FR (e.g., video surveillance) is 10 to 20 pixels in the eye distance and face region is usually 1/16 th of commonly used TV recording resolution of 240 320 × pixels. Further, FRVT 2000 [3] studied the effect of resolution on performance until eye distance is as low as 5 5 × pixels. In addition, previous works examined how low-resolution gray-scale (or intensity) faces affect recognition performance. They revealed that much lower resolution faces significantly deteriorated the recognition performance comparing with high-resolution ones [3], [6-7]. Evidently, low-resolution faces impose a significant restriction on the conventional intensity-based FR systems to guarantee reliability and feasiability. Traditional resolution enhancement techniques such as ‘super-resolution’ could be used to handle low- resolution faces. One significant disadvantage, however, is that these techniques require multiple low-resolution facial images that belong to the same identity captured from the same scene. In practice, it is usually difficult to support such requirement (e.g., only a single face image of being the same person is usually available for annotation of faces on personal photos). To circumvent the problem due to low-resolution faces, selecting face features robust to changes in face resolution is critically important. In contrast to the intensity-driven features, color cue is known to be less susceptible to resolution changes [8]. In addition, the psychophysical result of the FR test in Human Visual System (HVS) showed that the contribution of color cue would be more evident when the shapes of face were degraded [9]. Usefulness of color in the computerized FR systems was demonstrated in many literatures [10-13]. They mainly focused on following issues: was color information helpful to improve the performance in comparison to gray-scale only [10-13]; how three spectral channels of color were utilized [10], [12]; which color space was the best [11-12]. However, color effect on face resolution has not yet been fully investigated in the current color-based FR works and Tenth IEEE International Symposium on Multimedia 978-0-7695-3454-1/08 $25.00 © 2008 IEEE DOI 10.1109/ISM.2008.22 294