Face Recognition with Local Binary Patterns, Spatial Pyramid Histograms and Naive Bayes Nearest Neighbor classification Daniel Maturana, Domingo Mery and ´ Alvaro Soto Departamento de Ciencias de la Computaci´ on Pontificia Universidad Cat´ olica Santiago, Chile Email: {dimatura, dmery, asoto}@uc.cl Abstract—Face recognition algorithms commonly assume that face images are well aligned and have a similar pose – yet in many practical applications it is impossible to meet these conditions. Therefore extending face recognition to un- constrained face images has become an active area of research. To this end, histograms of Local Binary Patterns (LBP) have proven to be highly discriminative descriptors for face recognition. Nonetheless, most LBP-based algorithms use a rigid descriptor matching strategy that is not robust against pose variation and misalignment. We propose two algorithms for face recognition that are de- signed to deal with pose variations and misalignment. We also incorporate an illumination normalization step that increases robustness against lighting variations. The proposed algorithms use descriptors based on histograms of LBP and perform descriptor matching with spatial pyramid matching (SPM) and Naive Bayes Nearest Neighbor (NBNN), respectively. Our con- tribution is the inclusion of flexible spatial matching schemes that use an image-to-class relation to provide an improved robustness with respect to intra-class variations. We compare the accuracy of the proposed algorithms against Ahonen’s original LBP-based face recognition system and two baseline holistic classifiers on four standard datasets. Our results indicate that the algorithm based on NBNN outperforms the other solutions, and does so more markedly in presence of pose variations. Keywords-face recognition; local binary patterns; naive Bayes; nearest neighbor; spatial pyramid. I. I NTRODUCTION Most face recognition algorithms are designed to work best with well aligned, well illuminated, and frontal pose face images. In many possible applications, however, it is not possible to meet these conditions. Some examples are surveillance, automatic tagging, and human robot interac- tion. Therefore, there have been many recent efforts to develop algorithms that perform well with unconstrained face images [1]–[4]. In this context, the of use local appearance descriptors such as Gabor jets [5], [6], SURF [7], SIFT [8], [9], HOG [10] and histograms of Local Binary Patterns [11] have become increasingly common. Algorithms that use local appearance descriptors are more robust against occlusion, expression variation, pose variation and small sample sizes than traditional holistic algorithms [4], [5]. In this work we will focus on descriptors based on Local Binary Patterns (LBP), as they are simple, computationally efficient and have proved to be highly effective features for face recognition [3], [4], [12], [13]. Nonetheless, the methods described in this paper can be readily adapted to operate with alternative local descriptors. Within LBP-based algorithms, most of the face recogni- tion algorithms using LBP follow the approach proposed by Ahonen et al in [12]. In this approach the face image is divided into a grid of small of non overlapping regions, where a histogram of the LBP for each region is constructed. The similarity of two images is then computed by summing the similarity of histograms from corresponding regions. One drawback of the previous method is that it assumes that a given image region corresponds to the same part of the face in all the faces in the dataset. This is only possible if the face images are fully frontal, scaled, and aligned properly. In addition, while LBP are invariant against monotonic gray- scale transformations, they are still affected by illumination changes that induce non monotonic gray-scale changes such as self shadowing [17]. In this paper, we propose and compare two algorithms for face recognition that are specially designed to deal with moderate pose variations and misaligned faces. These algorithms are based on previous techniques from the object recognition literature: spatial pyramid matching [14], [15] and Naive Bayes Nearest Neighbors (NBNN) [16]. Our main contribution is the inclusion of flexible spatial match- ing schemes based on an “image-to-class” relation which provides an improved robustness with respect to intra-class variations. These matching schemes use spatially dependent variations of the “bag of words” models with LBP histogram descriptors. As a further refinement, we also incorporate a state of the art illumination compensation algorithm to improve robustness against illumination changes [17]. This paper is organized as follows. Section II discusses the details of our approach. Section III-C shows the results of applying our methodology to standard datasets. Finally, section IV presents the main conclusions of this work.