Considerations for emotion-aware consumer products Egon L. van den Broek a , Joyce H.D.M. Westerink b, * a Center for Telematics and InformationTechnology (CTIT), University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands b Philips Research, High Tech Campus 34, 5656 AE Eindhoven, The Netherlands article info Article history: Received 1 February 2008 Accepted 30 April 2009 Keywords: Short-term emotion assessment Physiological signals Statistical moments abstract Emotion-aware consumer products require reliable, short-term emotion assessment (i.e., unobtrusive, robust, and lacking calibration). To explore the feasibility of this, an experiment was conducted where the galvanic skin response (GSR) and three electromyography (EMG) signals (frontalis, corrugator supercilii, and zygomaticus major) were recorded on 24 participants who watched eight 2-min emotion inducing film fragments. The unfiltered psychophysiological signals were processed and six statistical parameters (i.e., mean, absolute deviation, standard deviation, variance, skewness, and kurtosis) were derived for each 10-s interval of the film fragment. For each physiological signal, skewness and kurtosis discriminated among affective states, accompanied by other parameters, depending on the signal. The skewness parameter also showed to indicate mixed emotions. Moreover, a mapping of events in the fragments on the signals showed the importance of short-term emotion assessment. Hence, this research identified generic features, denoted important considerations, and illustrated the feasibility of emotion- aware consumer products. Ó 2009 Elsevier Ltd. All rights reserved. 1. Introduction There is a growing interest in systems that are aware of user’s emotions. Such systems find their application in professional or specialized domains, such as emotional support for autistic persons (Teeters et al., 2006), stress in ambulance dispatchers (Mulder et al., 2004), irritation detection to support call center employees (Dev- illers et al., 2003), as therapy progress indicator for psychologists (Van den Broek, 2004), or with pilots and airline crews (Trimmel et al., 2005). In a typical consumer context, however, there is no explicit task at hand and the main intention is to support a pleasant everyday life. In such a context, empathic systems can adapt the conversational dialogue in order to optimize human–product interaction, can characterize someone’s emotional state for increased self-awareness or to others for enhanced communica- tion, or they can adapt the user’s environment to the present mood. The consumer context poses a number of boundary conditions that might be different with respect to the professional context (Chau and Hu, 2001; Wixom and Todd, 2005). A first distinction is in the accuracy required for emotion detection. Though any consumer or professional application would preferably comprise a flawless emotion awareness system, it is likely that every now and then the emotion detection is incorrect. It is to be expected that such errors are more detrimental in a professional application than in a consumer application, since they interfere with the profes- sional task. In many consumer applications, such a task is often less prominent or even absent, and the system’s reactions are not rigidly classified as correct of wrong, but rather as more or less preferred. Thus, a consumer application is somewhat more resilient with respect to emotion misclassifications, and most probably a higher percentage of them will be acceptable. A second point of difference pertains to the unobtrusiveness of the application. If one wears a system either for professional use or to compensate for a certain handicap, one will more easily accept that the actual use of the system is a hassle (Legris et al., 2003; Venkatesh, 2000; Wixom and Todd, 2005). For a consumer system, however, the emotion awareness system should preferably be unnoticeable to the user, the ultimate perceived ease of use. For instance, it could work from a distance, such as in speech or video processing. There, the detection of emotional features in the speech spectrum or in the facial expression (Cowie et al., 2001; Den Uyl and Van Kuilenberg, 2005; Van den Broek, 2004) can be done even without awareness of the user; however, the physical range in which these systems work is limited. To overcome these range problems, the system could be worn. Then, it is important that it is not noticeable to the user on a continuous basis. Another form of obtrusiveness is when the system needs constant (re-)calibration. Where a professional application can require regular calibrations in * Corresponding author. Tel.: þ31 40 2747793. E-mail addresses: vandenbroek@acm.org (E.L. van den Broek), joyce.westerink@ philips.com (J.H.D.M. Westerink). Contents lists available at ScienceDirect Applied Ergonomics journal homepage: www.elsevier.com/locate/apergo 0003-6870/$ – see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.apergo.2009.04.012 Applied Ergonomics 40 (2009) 1055–1064