Synthesis and Recognition of Face Profiles Frank Wallhoff and Gerhard Rigoll Technische Universit¨ at M¨ unchen, Institute for Human-Machine-Interaction Arcisstraße 21, 80290 M¨ unchen, Germany Email: {wallhoff,rigoll}@mmk.ei.tum.de Abstract Research on biometrical systems and especially on face recognition systems has become of high interest. Nowadays several approaches exist to recognize frontal views of faces. Under cer- tain constraints the recognition accuracy even for huge databases seems to be acceptable. How- ever, it has shown, that the recognition performance of nearly all state-of-the-art systems dramatically drops down, when faces rotated in-depth or even profile views are presented. In this work we present our actual experiments and results of neural network based approaches for face profile recognition systems. One of the main challenges is to implement a self learning approach that does not need any direct 3D information. The key idea of our approaches is to derive the corre- spondences between head profiles and frontal views by examples automatically. With this approach we are able to synthesize profile views of heads by pre- senting frontal views. The quality of the result- ing systems can be measured with Hidden Markov Models (HMM) and an extension to the well-known Eigenfaces approach. The performances are evalu- ated on the MUGSHOT and the FERET database. 1 Introduction Face recognition technology (FERET as introduced by the US Army Research Laboratory, ARL) has become increasingly important for several fields of applications, such as controlling who is entering a building (access control) or detection of violent criminals and terrorists in airports or other public places. Although many different approaches have been presented for the frontal face recognition prob- lem [1, 2], it seems not to be solved yet for real world applications on huge databases, as also stated in [3] and [4]. Especially the recognition of faces rotated in depth is not generally solved. Therefore this paper addresses our actual re- search on recognizing synthesized faces rotated by 90 degrees in depth with neural and statistical ap- proaches. When we started our work on this chal- lenging task, we noticed, that there has not been done much investigation in this area before. We found several approaches, which concentrated on generating synthesized 3D images of faces using 3D data [5, 6]. Other, newer approaches are mainly based on so called head meshes or wire-frames to model 3D information into planar images [7, 8]. However, all the approaches above make use of expensive gathered or labeled 3D-data. Our goal is to engineer a self-learning approach using suffi- cient planar training material for the synthesis in- stead. For this task we deploy a neural network to learn the rotation process from training examples. Another property of such an example is that it can synthesize regions of the profile that were occluded in the frontal view by the use of learned knowledge such as the hair behind the ears. To classify the quality of the profile views, we examined HMM based classification approaches and an extension of the well-known Eigenface ap- proach. The paper is structured as follows. Two different approaches for face profile synthesis using neural networks are introduced in the successive section, which is followed by the classification techniques using HMMs and an Eigenmugshots. In the next section the obeyed databases are briefly introduced. Hereafter the classification results of the presented systems are discussed. The paper closes with a sum- mary and outlook. VMV 2003 Munich, Germany, November 19–21, 2003