How to deceive a face recognizer? B. G ¨ okberk, L. Akarun, B. Aksan Computer Engineering Dept., Bo ˘ gazic ¸i University {gokberk, akarun, aksan}@boun.edu.tr Abstract Many security systems depend upon face recognizers to identify a person. Many of these systems are passive and are deployed at places such as airline terminals. However, face recognizers are sensitive to deception attacks. Previ- ous studies suggest that hair regions are very crucial in face recognition and the success of a recognizer depends on the success of a pre-segmentation stage which extracts the face region from the hair and the background. Deception attacks which would change the hairstyle, apply make-up or occlud- ing objects to the face would cause many systems to fail. In this study, we study the effects of deception attacks on two basic face recognition systems: a PCA-based system and a Gabor wavelet-based recognizer. We study the performance of the recognizers under different attacks and focus on the selection of features so as to maximize performance under attacks. 1. Introduction Over the past two decades significant progress has been made in the automatic human face recognition research. Al- though many successful face recognition systems have been proposed in the literature, the problem is still not consid- ered to be fully solved, especially in real-life applications. The main obstacle can be simply stated as follows: intra- personal variations between human faces is large when compared to inter-personal variations. These variations can be broadly classified into two groups: external variations and internal variations. Variations due to illumination, head pose, scale and translation are considered to be external variations. However, variations due to hair color, hair style, moustache, beard and eyeglasses as well as facial variations which stem from the subject itself are considered to be in- ternal variations. One of the studies dealing with internal variations such as expression changes and occlusion is [1], where the AR face database is used to illustrate the superior performance of a local probabilistic approach. The local component based approach has also been studied to deal with external variations in face recognition [2, 3]. When occlusions such as beards and glasses are present, a different approach is to try to remove them [4, 5]. In this paper, our aim is to examine how an impostor can deceive a face recognizer by taking the advantage of in- ternal variations; specifically hair color change, occlusions, and expression variations. After analyzing the effects of such variations aiming to deceive a recognizer, we propose a robust technique that increases the performance of PCA and Gabor-based face recognizers. 2. Face Representation 2.1. PCA-based Method In PCA, faces are expressed as linear combinations of the eigenvectors of faces. Then, for recognition, the PCA coefficients can be used to denote a face. In its original form, PCA is found to be rather sensitive to image inten- sity variations, local perturbations, and needs almost perfect correspondence. Image variations which are not present in the training phase generally cause a poor recognition perfor- mance. A possible solution to improve the PCA method is to divide the whole face region into subregions and do mod- ular PCA analysis. In a modular PCA analysis, each subre- gion is handled in isolation, and for each subregion, a dif- ferent subspace is found. Then local features are extracted and merged to represent a face. An important advantage of modular PCA analysis is that local perturbations can only affect the local coefficients, not the whole face. Figure 1.c shows subregions that we have used in our experiments. 2.2. 2D Gabor Wavelet-based Method A biologically motivated representation of face images is to code them using convolutions with multi-frequency multi-orientation 2D Gabor–like filters. In order to repre- sent face images using Gabor filters, the intensity image is convolved by Gabor kernels. The set of convolution coef- ficients for kernels of different orientations and frequencies 1