682 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 12, NO. 7, NOVEMBER 2010 A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference Shangfei Wang, Member, IEEE, Zhilei Liu, Siliang Lv, Yanpeng Lv, Guobing Wu, Peng Peng, Fei Chen, and Xufa Wang Abstract—To date, most facial expression analysis has been based on visible and posed expression databases. Visible images, however, are easily affected by illumination variations, while posed expressions differ in appearance and timing from natural ones. In this paper, we propose and establish a natural visible and infrared facial expression database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible and an infrared thermal camera, with illumination provided from three different direc- tions. The posed database includes the apex expressional images with and without glasses. As an elementary assessment of the usability of our spontaneous database for expression recogni- tion and emotion inference, we conduct visible facial expression recognition using four typical methods, including the eigenface approach [principle component analysis (PCA)], the fisherface approach [PCA + linear discriminant analysis (LDA)], the Active Appearance Model (AAM), and the AAM-based + LDA. We also use PCA and PCA+LDA to recognize expressions from infrared thermal images. In addition, we analyze the relationship between facial temperature and emotion through statistical analysis. Our database is available for research purposes. Index Terms—Emotion inference, expression recognition, facial expression, infrared image, spontaneous database, visible image. I. INTRODUCTION F ACIAL expression is a convenient way for humans to communicate emotion. As a result, research on expres- sion recognition has become a key focus area of personalized human-computer interaction [1], [2]. Most current research focuses on visible images or videos and good performance has been achieved in this regard. Whereas varying light exposure can hinder visible expression recognition, infrared thermal images, recording the temperature distribution formed by face vein branches, are not sensitive to imaging conditions. Thus, thermal expression recognition is a useful and necessary com- plement to visible expression recognition [3]. Besides, a change Manuscript received December 13, 2009; revised March 24, 2010 and June 23, 2010; accepted June 24, 2010. Date of publication July 26, 2010; date of current version October 15, 2010. This paper is supported in part by National 863 Program (2008AA01Z122), in part by Anhui Provincial Natural Science Foundation (No.070412056), and in part bySRF for ROCS, SEM. The associate editor coordinating the review of this manuscript and approving it for publica- tion was Dr. Caifeng Shan. The authors are with the School of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, China (e-mail: sfwang@ustc.edu.cn; leivo@mail.ustc.edu.cn; lsliang@mail.ustc.edu.cn; lvyp@mail.ustc.edu.cn; guobing@mail.ustc.edu.cn; dbpeng@mail.ustc. edu.cn; feichen@mail.ustc.edu.cn; xfwang@ustc.edu.cn). Digital Object Identifier 10.1109/TMM.2010.2060716 in facial temperature is a clue that can prove helpful in emotion inference [4], [5]. Furthermore, most existing research has been based on posed expression databases, which are elicited by asking subjects to perform a series of emotional expressions in front of a camera. These artificial expressions are usually exaggerated. Spontaneous expressions, on the other hand, may be subtle and differ from posed ones both in appearance and timing. It is, therefore, most important to establish a natural database to allow research to move from artificial to natural expression recognition, ultimately leading to more practical applications thereof. This paper proposes and establishes a natural visible and in- frared facial expression database (NVIE) for expression recog- nition and emotion inference. First, we describe in detail the de- sign, collection, and annotation of the NVIE database. In addi- tion, we conduct facial expression analysis on spontaneous vis- ible images with front lighting using several typical methods, including the eigenface approach [principle component anal- ysis (PCA)], the fisherface approach [PCA + linear discriminant analysis (LDA)], the Active Appearance Model (AAM), and the combined AAM-based + LDA (referred to as AAM+LDA). Thereafter, we use PCA and PCA +LDA to recognize expres- sions from spontaneous infrared thermal images. In addition, we analyze the relationship between facial temperature and emo- tion through an analysis of variance (ANOVA). The evaluation results verify the effectiveness of our spontaneous database for expression recognition and emotion inference. II. BRIEF REVIEW OF EXISTING NATURAL AND INFRARED DATABASES There are many existing databases dealing with facial ex- pressions, an exhaustive survey of which is given in [1] and [6]. Here, we only focus on natural and infrared facial expression databases. Due to the difficulty of eliciting affective displays and the time-consuming manual labeling of spontaneous ex- pressions, only a few natural visible expression databases exist. These are listed in Table I, together with details of size, elicitation method, illumination, expression descriptions, and modality [visual (V) or audiovisual (AV)]. From Table I, we can see that researchers use one of three possible approaches to obtain spontaneous affective behavior, i.e., human-human con- versation, human-computer interaction, or emotion-inducing videos [22]. Since this paper only focuses on facial expressions, and not speech or language, using emotion-inducing videos is 1520-9210/$26.00 © 2010 IEEE