684 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 3, NO. 4, DECEMBER 2 Recognizing Rotated Faces From Frontal and Views: An Approach Toward Effective Use o Mugshot Databases Xiaozheng Zhang, Student Member, IEEE,Yongsheng Gao, Senior Member, IEEE,and Maylor K. H. Leung, Member, IEEE Abstract—Mug shot photography has been used to identify criminals by the police for more than a century. However, the common scenario of face recognition using frontal and side-view mug shots as gallery remains largely uninvestigated in comput- erized face recognition across pose. This paper presents a novel appearance-based approach using frontal and sideface images to handle pose variations in face recognition, which has great potential in forensic and security applications involving police mugshot databases. Virtual views in different poses are generated in two steps: 1) shape modelling and 2) texture synthesis. In the shape modelling step, a multilevel variation minimization approach is applied to generate personalized 3-D face shapes. In the texture synthesis step, face surface properties are analyzed and virtual views in arbitrary viewing conditions are rendered, taking diffuse and specular reflections into account. Appear- ance-based face recognition is performed with the augmentation of synthesized virtual views covering possible viewing angles to recognize probe views in arbitrary conditions. The encouraging experimental results demonstrated that the proposed approach by using frontal and side-view images is a feasible and effective solution to recognizing rotated faces, which can lead to a better and practical use of existing forensic databases in computerized human face-recognition applications. Index Terms—Appearance-based recognition, face recognition, mug shot, police database, pose variation, virtual view synthesis, 3-D modelling. I. I NTRODUCTION F ACE recognition under pose variations is one of the key remaining problems in the research field of pattern recog- nition and computer vision [42], [62]. It is of great interest in many applications, most notably those dealing with indifferent or uncooperative subjects, for instance, in surveillance systems. Given the current state of technologies, however, computerized face recognition requires cooperative subjects who stay still in Manuscript received October 09, 2007; revised April 21, 2008. Current ver- sion published November 19, 2008. This work was supported by the Australian Research Council (ARC) under Discovery Grant DP0451091. The associate ed- itor coordinating the review of this manuscript and approving it for publication was Prof. Vijaya Kumar Bhagavatula. X. Zhang and Y. Gao are with the Griffith School of Engineering, Griffith University, Nathan Qld 4111, Australia (e-mail: x.zhang@griffith.edu.au; yong- sheng.gao@griffith.edu.au). M. K. H. Leung is with the School of Computer Engineering, Nanyang Tech- nological University, Singapore 639798 (e-mail: asmkleung@ntu.edu.sg). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TIFS.2008.2004286 a required pose (often frontal) to be checked [9]. The diffi- culty lies in the fact that the intraclass differences brough pose variations are often larger than interclass differences distinguishing different people. One of the most successfu proaches for face recognition is the appearance-based app [27]. In a pose-invariant face-recognition scenario, it comp an input (or probe) face in an arbitrary pose with a numbe enrolled (or gallery) images per person in the database co ering all possible viewing directions. The performance of ap- pearance-based face recognition is largely dependent on t lection and availability of the gallery images. Police mugshot databases 1 usually consist of a frontal view and a side view per person (Fig. 1), which are a primary s of gallery images for face recognition [55]. The police hav mug shots to identify criminals for more than a century [4 and there are many legacy mugshot databases which were tinely taken during the police booking process [31]. For co puterized face recognition, however, frontal and side-view shots are not able to provide effective coverage of all poss conditions, when a probe image is in an arbitrary pose bet the frontal and side-viewing angles. Although several pose variant face-recognition techniques [7], [24],[27] have been proposed recently, the common scenario of using frontal and side-view mug shots as a gallery remains largely uninvesti This research proposes a novel face-recognition approach uses frontal and side-view face images as gallery and recog- nizes probe views in arbitrary poses, which has great poten- tial in mugshot-related applications. The combination of fr view and side-view balances the tractability and applicabil of the proposed approach for face recognition across pose Compared to algorithms of face recognition from single im multiple gallery views provide more information about the dividual face so that the system can be more accurate and less on prior knowledge of an average human face. The pro- posed approach aims to make effective use of the frontal a side views of face pictures widely available in existing pol mugshot databases and does not require additional acquis of gallery databases. It can easily fit into many face-recogn applications using frontal and side-view mug shots as a ga for law enforcement and security surveillance, for exampl video-monitoring systems. The proposed approach is appearance based, which synt sizes personalized virtual views in different poses from mu 1 Mugshot databases in this paper refer to common police databases con taining frontal and side views. 1556-6013/$25.00 © 2008 IEEE