Gathering Emotional Data from Multiple Sources Sergio Salmeron-Majadas aDeNu Research Group. UNED Calle Juan del Rosal, 16. Madrid 28040. Spain +34 91 398 93 88 ssalmeron@bec.uned.es Olga C. Santos aDeNu Research Group. UNED Calle Juan del Rosal, 16. Madrid 28040. Spain +34 91 398 93 88 ocsantos@dia.uned.es Jesus G. Boticario aDeNu Research Group. UNED Calle Juan del Rosal, 16. Madrid 28040. Spain +34 91 398 91 97 jgb@dia.uned.es Raúl Cabestrero UNED Calle Juan del Rosal, 10. Madrid 28040. Spain +34 91 398 62 40 rcabestrero@psi.uned.es Pilar Quirós UNED Calle Juan del Rosal, 10. Madrid 28040. Spain +34 91 398 82 48 pquiros@psi.uned.es Mar Saneiro aDeNu Research Group. UNED Calle Juan del Rosal, 16. Madrid 28040. Spain +34 91 398 91 98 marsanerio@dia.uned.es ABSTRACT Collecting and processing data in order to detect and recognize emotions has become a research hot topic in educational scenarios. We have followed a multimodal approach to collect and process data from different sources to support emotion detection and recognition. To illustrate the approach, in this demo, participants will be shown what emotional data can be gathered while solving Math problems. Keywords Affective Computing, Data Mining, Sensor Data, Emotion Detection, Mathematics 1. INTRODUCTION Currently there is a growing interest in offering emotional support to learners in e-learning platforms through an expanded set of adaptive features. A key issue is to determine learners’ affective state, which is related to their cognitive and metacognitive process [4], preferable with low cost sensors [2]. Affective states in our approach are to be defined from mining in a jointly manner subjective, physiological and behavioral data gathered from diverse emotional information sources while the learner interacts on the given e-learning environment. This approach offers possible improvements on emotion detection, which as suggested in the literature may come out from the combination of different data sources simultaneously [5]. Math problem solving scenarios have provided opportunities to investigate this new approach, as from them different emotions may be elicited [7]. 2. OUR APPROACH As for emotion detection, our approach is based on the use of data mining techniques. As shown in Figure 1, we follow a multimodal gathering approach based on the combination of the following data sources obtained while the learner carries out learning interactions to solve Mathematical tasks in the e- learning platform and stored in the corresponding user model. To start with, bio-feedback data provide appropriate measures to detect typical physiological reactions that come along with emotions. Although they should not be used for categorizing discrete emotions on its own, they provide useful indicators of the participants’ arousal level associated with the ongoing affective state over the learning process. Signals used to this end are: heart rate, breath frequency, galvanic skin response and skin temperature. To evaluate phasic variations on collected signals upon a tonic state, recordings of each learner pre- baseline are done to provide reference values for subsequent analysis. Another key source for gathering affective information is the non-verbal behavior (e.g. gestures, facial expression, body movements). Facial expressions of participants are recorded by Windows Kinect face features extraction. Kinect for Windows device provides an API able to detect a user’s face model based in 100 points. The processing of these data is to identify the learner’s head position, inclination and expressions. In addition, a webcam (with microphone) is used to record other sources of information not necessarily located in the participant’s face expression, such as verbal expression and speech tone. Some additional user interactions are also gathered. In particular, keyboard and mouse data sources are recorded to find out behavioral correlates of the emotional intensity. To collect all the events triggered by mouse and keyboard, a key logger and mouse tracker has been developed in Java (with no GUI, so it cannot interfere with the user interactions) using the library provided by kSquared.de. A video of participant’s desktop is also recorded to keep track of the session. As this approach is based on a wide range of information sources, synchronization is a key issue. Due to the number of devices used, several computers may be needed to collect the required information. Thus, the synchronization of the systems involved (given that some of the recorded interactions can last less than a second) is needed. Through this, synchronization data are merged and data mining can be applied. Information about learners’ personality is also considered when processing the data, given the narrow relation between personality traits and affective states with learning styles and