Article DOI: 10.1111/exsy.12014 Electrocardiogram-based emotion recognition system using empirical mode decomposition and discrete Fourier transform Jerritta S, * M Murugappan, Khairunizam Wan, Sazali Yaacob School of Mechatronic Engineering, Universiti Malaysia Perlis (UniMAP), Ulu Pauh, Arau, Perlis 02600, Malaysia E-mail: sn.jerritta@gmail.com Abstract: Emotion recognition using physiological signals has gained momentum in the field of human computer–interaction. This work focuses on developing a user-independent emotion recognition system that would classify five emotions (happiness, sadness, fear, surprise and disgust) and neutral state. The various stages such as design of emotion elicitation protocol, data acquisition, pre-processing, feature extraction and classification are discussed. Emotional data were obtained from 30 undergraduate students by using emotional video clips. Power and entropy features were obtained in three ways – by decomposing and reconstructing the signal using empirical mode decomposition, by using a Hilbert–Huang transform and by applying a discrete Fourier transform to the intrinsic mode functions (IMFs). Statistical analysis using analysis of variance indicates significant differences among the six emotional states (p < 0.001). Classification results indicate that applying the discrete Fourier transform instead of the Hilbert transform to the IMFs provides comparatively better accuracy for all the six classes with an overall accuracy of 52%. Although the accuracy is less, it reveals the possibility of developing a system that could identify the six emotional states in a user-independent manner using electrocardiogram signals. The accuracy of the system can be improved by investigating the power and entropy of the individual IMFs. Keywords: human–computer interaction; emotion recognition; empirical mode decomposition; electrocardiogram (ECG) 1. Introduction Over the last decade, researchers on human–computer interaction have focussed their attention on empowering computers with some amount of emotional intelligence. Emotional intelligence is the ability to recognize, express and have emotions, coupled with an ability to regulate and harness them for constructive purposes (Picard et al., 2001). The addition of emotional intelligence to systems and computers will play a significant role not only in designing ‘intelligent rooms’ (Hirsh et al., 1999) and ‘affective tutoring’ (Rosalind, 2003) but also in developing the next-generation robots (Arkin et al., 2001). By studying the various ways of expressing emotions, emotional intelligence has been endowed on systems using several modalities. Methods of recognizing emotion using facial expressions and speech signals have gained much inter- est and achieved better results (Ang et al., 2004; Bailenson et al., 2008; Kessous et al., 2009). Yet these modalities necessitate the emotion to be externally expressed either by facial actions or by speaking. Socially masked emotions, unexpressed emotions or minor emotional changes that cannot be perceived externally, cannot be tracked using these modalities. These limitations paved the way to recognizing emotions through physiological responses (Kim et al., 2004). As physiological signals reflect the inherent activity of the autonomous nervous system, social masking does not have an impact in recognizing the true emotions felt by the person. This also provides an opportunity to track minor emotional changes that cannot be perceived on seeing or hearing a person (Rani and Sarkar, 2006). Biosignals such as electroencephalogram, electrocardio- gram (ECG), electromyogram, galvanic skin response, skin temperature, blood volume pulse and respiratory response have been used by various researchers to track emotional changes (André et al., 2004; Kim et al., 2004; Korb et al., 2008). In this paper, we have used ECG signals to study six emotional state changes (happiness, sadness, fear, surprise, disgust and neutral) in the subjects. We have proposed an algorithm combining empirical mode decomposition (EMD) and Fourier spectral analysis for better emotion recognition. This paper is organized as follows. Section 1.1 defines the various emotional states. Section 2 describes the method- ology, providing insight to the various stages such as data acquisition, pre-processing, feature extraction and classifi- cation. Section 3 consolidates the results and discusses the research findings. Section 4 concludes with a remark on further enhancement in emotion classification. 1.1. Emotional states Emotions can be defined as a complex psychophysiological experience of an individual’s state of mind. It has been studied in various fields such as psychology, cognitive science, philoso- phy and computer science. Researchers across these disciplines have agreed on two basic emotional models – the discrete model and the dimensional model (Ekman et al., 1983). The dimensional model specifies emotions on the basis of two scales – valence and arousal. The polarity (positive or negative) and intensity of emotions are captured by the valence and arousal scales, respectively (Lang, 1995). The discrete emotional model indicates the universal presence of six basic emotions, namely happiness, sadness, fear, surprise, disgust and anger (Ekman and Friesen, 1987). Figure 1 shows the basic emotions plotted on the valence–arousal plane (Lang, 1995). © 2013 Wiley Publishing Ltd 110 Expert Systems, May 2014, Vol. 31, No. 2