IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 14, NO. 3, MAY 2010 589 Toward Emotion Aware Computing: An Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli Christos A. Frantzidis, Charalampos Bratsas, Member, IEEE, Christos L. Papadelis, Evdokimos Konstantinidis, Costas Pappas, and Panagiotis D. Bamidis, Member, IEEE Abstract—This paper proposes a methodology for the robust classification of neurophysiological data into four emotional states collected during passive viewing of emotional evocative pictures selected from the International Affective Picture System. The pro- posed classification model is formed according to the current neu- roscience trends, since it adopts the independency of two emotional dimensions, namely arousal and valence, as dictated by the bidirec- tional emotion theory, whereas it is gender-specific. A two-step clas- sification procedure is proposed for the discrimination of emotional states between EEG signals evoked by pleasant and unpleasant stimuli, which also vary in their arousal/intensity levels. The first classification level involves the arousal discrimination. The valence discrimination is then performed. The Mahalanobis (MD) distance- based classifier and support vector machines (SVMs) were used for the discrimination of emotions. The achieved overall classification rates were 79.5% and 81.3% for the MD and SVM, respectively, sig- nificantly higher than in previous studies. The robust classification of objective emotional measures is the first step toward numerous applications within the sphere of human–computer interaction. Index Terms—Affective computing, EEG, emotion theory, human–computer interaction (HCI), Mahalanobis, neurophysio- logical recordings, support vector machines (SVMs). I. INTRODUCTION D URING the past few years, there has been a rapid develop- ment in the field of human–computer interaction (HCI). New methodologies in user interfaces were introduced aim- ing to improve the interaction between the human and the ma- chine [1], [2]. Nowadays, computers are no longer viewed as merely computational tools, but as the enabling technology for new ways of cyber communication [3], cyber relationships [4], and cyber socialization [5], since they are equipped with all the necessary communication channels to interact with the users, Manuscript received July 10, 2009; revised October 23, 2009. First published February 17, 2010; current version published June 3, 2010. This work was supported by a grant from the Greek General Secretariat for Research and Technology and by the M.Sc. in Medical Informatics Programme, Aristotle University of Thessaloniki, Greece. C. A. Frantzidis, C. Bratsas, E. Konstantinidis, C. Pappas, and P. D. Bamidis are with the Laboratory of Medical Informatics, Medical School, Aristotle University of Thessaloniki, Thessaloniki 54124, Greece (e-mail: christos. frantzidis@gmail.com; mpampis@med.auth.gr; evdokimosk@gmail.com; cpappas@med.auth.gr; bamidis@med.auth.gr). C. L. Papadelis is with the Center for Mind/Brain, University of Trento, Trento 38068, Italy (e-mail: christos.papadelis@unitn.it). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TITB.2010.2041553 and the sensing abilities to infer user’s attributes [1]. It is there- fore imperative and unsurprising that computerized systems rapidly prevail in various socioemotional life aspects, such as e-health, education, telemonitoring of elderly people, and learn- ing [3], [6], [7]. There are however many challenges that should be faced toward the achievement of successful HCI. Among them, the increasing heterogeneity of computer users, and the stricter demands for reduced user frustration while interacting with computers, which both are causing a shift of the adaptation requirements from the human user to the computer [8]. To be more realistic and robust, the HCI should follow the basic principles of communication among human beings [9]. Thus, machines should be equipped with socioemotional skills in order to naturally adapt to their users’ emotional mood ex- pressing feelings, such as frustration, confusion, disliking, or interest [10]. These arguments put recently, alongside the signif- icance of embedding emotion recognition to HCI systems, gave birth to a newly introduced facet of human intelligence named as “emotional intelligence” within the more general field of ambient intelligence (AmI) [11]. The core element of emotional intelligence is one’s ability to recognize the affective state of the person communicating with, so as to alter his/her own behavior according to this information. Embedding a subset of human emotional skills to machines will lead to the next generation of HCI technology, which will facilitate computers with the abil- ity to adapt their function in a more human-like, effective, and efficient way [1]. But before dealing with emotion aware computing, a pre- requisite of vital importance is to meet the theoretical assump- tions associated with human emotion and emotional reactions. According to a currently widely accepted view, a simple but yet effective theory [12], [13], employed a bidirectional emo- tional model defined by two basic parameters/variables, namely, the affective degree of pleasantness and arousal. The model re- gards the various emotional states as being subordinate divisions placed in a 2-D emotional space formed by these two affective variables. Its divisions are correlated to the species survival, since emotions are based on either an appetitive or a defensive motivational system, according to the judged degree of pleas- antness/unpleasantness it elicits (valence dimension). The acti- vation level of the active motivational system defines the arousal dimension, whether the emotional state is regarded as calm or exciting [14]. To standardize an experimental framework for human emotions’ research, the International Affective Picture System (IAPS) was developed by Lang [15]. The IAPS system 1089-7771/$26.00 © 2010 IEEE Authorized licensed use limited to: Sheffield University. Downloaded on June 02,2010 at 11:43:55 UTC from IEEE Xplore. Restrictions apply.