Integrated 3D Expression Recognition and Face Recognition Chao Li, Armando Barreto Electrical & Computer Engineering Department Florida International University Miami, Florida, 33174, USA {cli007, barretoa}@fiu.edu http://dsplab.eng.fiu.edu Abstract Face recognition technology has been a focus both in academia and industry for the last couple of years because of its wide potential applications and its importance to meet the security needs of today’s world. Most of the systems developed are based on 2D face recognition technology, which uses pictures for data processing. With the development of 3D imaging technology, 3D face recognition emerges as an alternative to overcome the difficulties inherent with 2D face recognition, i.e. sensitivity to illumination conditions and positions of a subject. But 3D face recognition still needs to tackle the problem of deformation of facial geometry that results from the expression changes of a subject. To deal with this issue, a 3D face recognition framework is proposed in this paper. It is composed of three subsystems: expression recognition system, expressional face recognition system and neutral face recognition system. A system for the recognition of faces with one type of expression (smile) and neutral faces was implemented and tested on a database of 30 subjects. The results proved the feasibility of this framework. 1. Introduction Face recognition, together with fingerprint recognition, speaker recognition, etc., is part of the research area known as ‘biometric identification’ or ‘biometrics’, which refers to identifying an individual based on his or her distinguishing characteristics. Face recognition is a particularly compelling biometric approach because it is the one used every day by nearly everyone as the primary means for recognition of other humans. Because of its natural character, face recognition is more acceptable than most other biometric techniques. Face recognition also has the advantage of being noninvasive. Face recognition has a wide range of potential applications for commercial, security, and forensic purposes. These applications include automated crowd surveillance, access control, credit card authorization, design of human computer interfaces, etc. Especially, the surveillance systems rely on the noninvasiveness of face recognition systems. Face recognition approaches can be divided according to the format of the data acquired into 2D face recognition, 3D face recognition and infrared face recognition modalities. The infrared face recognition is commonly combined with other biometric technologies. Most of the face recognition attempts that have been made until recently use 2D intensity images by photographic cameras as the data format for processing. This kind of research is called 2D face recognition. Varying levels of success have been achieved in 2D face recognition research. Detailed and comprehensive surveys can be found in [1, 2]. Although 2D face recognition has achieved considerable success, certain problems still exist. Because the 2D face images used not only depend on the face of a subject, but also depend on the imaging factors, such as the environmental illumination and the orientation of the subject. These two sources of variability in the face image often make the 2D face recognition system fail. That is the reason why 3D face recognition is believed to have an advantage over 2D face recognition. With the development of 3D imaging technology, more and more attention has been directed to 3D face recognition. In [3], Bowyer et al. provide a survey of 3D face recognition technology. Some of the techniques are derived from 2D face recognition, such as PCA used in [4, 5]to extract features from faces. Some of the techniques are unique to 3D face recognition, such as the geometry matching method in [6] and the profile matching proposed in [7, 8]. Most of the 3D face recognition systems treat the 3D face surface as a rigid surface. But actually, the face surface is deformed by different expressions of the subject. Therefore, systems that treat the face as a rigid surface are prone to fail when dealing with faces with expressions. In [9], experiments using Iterative Closest Point (ICP) and PCA methods were performed on the recognition of faces with expression. The authors found that expression changes do cause performance to deteriorate in all methods. Therefore, the involvement of facial expression has become a big challenge in 3D face recognition systems. Up to now, only some methods address the facial expression issue in face recognition. In [10], the authors