An Ensemble Method for Classifying Startle Eyeblink Modulation from High-Speed Video Records Reza R. Derakhshani, Member, IEEE, and Christopher T. Lovelace Abstract—Psychophysiological measurements of startle eyeblink can provide information about the state of an individual regarding sensory, attentional, cognitive, and affective processing, and thus reveal valences of interest for affective computing. However, eyeblink is usually measured using intrusive contact electromyographic (EMG) electrodes, accompanied by a laborious manual process of feature extraction. We introduce a new noninvasive automatic system using high-speed video recording of startle blinks in conjunction with data-driven feature selection and support vector machine (SVM) ensembles to classify startle eyeblinks. Using a prestimulus (prepulse) to produce robust modulation of acoustically elicited startle eyeblinks, we tracked the blinks using 250 frames per second video, and extracted different features from eyelid displacement and velocity signals. The SVMs were able to determine whether a trial had contained startle or prepulse+startle stimuli with an accuracy of up to 73 percent (five-fold cross validation). By fusing the decisions made on different feature sets, an ensemble of seven SVMs increased this rate to almost 79 percent. Since startle eyeblinks are robustly modulated by not only sensory events (such as the prepulse used in this study) but also affective and cognitive states, eyelid tracking using high-speed video, in conjunction with the introduced classification method, is an effective and user-friendly alternative to EMG for classification of startle blinks to infer users’ affective-cognitive states. Index Terms—Affective computing, image processing, pattern recognition, user interfaces. Ç 1 INTRODUCTION N UMEROUS advances have been made in the field of affective computing in particular and Human Com- puter Interaction (HCI) in general during the past couple of decades in order to enable computers to recognize emotions and respond accordingly [1], [2], [3]. The momentary affective state of the human user has been of considerable interest, but difficult to measure. In the field of psychology, emotional state is typically measured by asking a set of emotion-relevant questions; however, this necessitates interruption of the ongoing task. In the nonverbal domain, most psychophysiological computer emotion recognition research efforts have been focused on modalities such as facial expressions and thermography, Galvanic skin resis- tance, eye movement (mostly gaze focus and direction, as well as saccades), voice, and heart rate. The list of such endeavors is indeed very long, with hundreds of related publications quoted in recent surveys (e.g., see [4], [5]). Our aim is to present a novel measure that offers some advantages over these other methods. Given that we have no direct access to a person’s internal emotional state, this must be done using observable, measurable cues. Our measure is based on the startle reflex, which can be reliably elicited by a sudden, intense, and unexpected sensory event, such as a loud sound [6]. The withdrawal response thus induced serves to retract the limbs, close the eyes, and protect the body from damage. This protective response is of interest to us because it is reliably modulated by a person’s affective state [6], [7]. For example, the startle response elicited when an individual is looking at an unpleasant picture (e.g., a dismembered hand) will be larger than one elicited while viewing a neutral picture (e.g., a coffee cup), and the startle response is smaller when viewing a positive picture (e.g., puppies) compared to a neutral picture [8]. This consistent pattern of modulation makes the elicited startle response a sensitive measure of affective state. The size and timing of a startle response is typically evaluated by quantifying the reflexive eyeblink response [9] (Fig. 1), which is usually recorded using electromyographic (EMG) electrodes adhered to the skin just beneath the lower eyelid. While the EMG response is a sensitive index of startle eyeblink, this technique has the disadvantage of being fairly obtrusive, requiring cleaning of the skin and adhesion of electrodes. For some applications, such as everyday comput- ing where user convenience is of the essence, a noncontact technique is desired, that is, one where the eyeblink response could be measured without the need for electrodes to detect individuals’ affective-cognitive states of interest. To that end, we previously validated eyelid movement vectors extracted from high-speed (250 frames per second) video recordings as a measure of startle eyeblink modulation (SEM) [10], [11]. Our eventual aim is to develop sensitive and specific 50 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 2, NO. 1, JANUARY-JUNE 2011 . R.R. Derakhshani is with the Department of Computer Science and Electrical Engineering, School of Computing and Engineering, University of Missouri-Kansas City, Robert H. Flarsheim Hall, 5100 Rockhill Rd, Kansas City, Missouri 64110-2499. E-mail: reza@umkc.edu. . C.T. Lovelace is with the Department of Psychology, University of Missouri-Kansas City, 5100 Rockhill Rd, Kansas City, MO 64110. Manuscript received 7 Mar. 2010; revised 18 Oct. 2010; accepted 20 Oct. 2010; published online 30 Nov. 2010. Recommended for acceptance by S.-W. Lee. For information on obtaining reprints of this article, please send e-mail to: taffc@computer.org, and reference IEEECS Log Number TAFFC-2010-03-0016. Digital Object Identifier no. 10.1109/T-AFFC.2010.15. 1949-3045/11/$26.00 ß 2011 IEEE Published by the IEEE Computer Society