Employing Think-Aloud Protocol to Connect User Emotions and Mouse Movements Avar Pentel Institute of Informatics Tallinn University Tallinn, Estonia pentel@tlu.ee Abstract—This paper describes unobtrusive method for user confusion detection by monitoring mouse movements. A special computer game was designed to collect mouse logs. Think-aloud protocol was used in order to identify the states of confusion. Features extracted from mouse movement’s logs were used in training dataset. Support Vector Machines, Logistic Regression, C4.5 and Random Forest were used to build classification models. Model generated by Random Forest yield to best classification results with f-score 0.938. Keywords—affective computing; confusion detection; thinking aloud protocol; mouse dynamics; behavioral biometrics. I. INTRODUCTION The system’s ability to detect users emotional states, gives promising applications to adaptive recommendations, e- learning systems, adaptive interfaces, etc. There are many measurable physiological characteristics that are related to human emotions. Using electroencephalogram, skin conductance and blood volume pressure measurements [1], [2] or detection of gaze and facial expressions [3], [4] proved to be successful techniques in emotion detection. Despite of success, these techniques are difficult to apply in real life. Therefore unobtrusive standard input devices like mouse and keyboard still remain the applicable candidates for real life applications. In the current study, we give a short overview about other related works and about one of our previous work [5] on confusion detection, and address some limitations that might be present in previous work. Then we present our new confusion detection approach. A. Related Works The theory of “embodied cognition” [6] gives a theoretical framework studying mouse movements in order to predict mental states. Barsalou suggests that this bi-directional relationship between mental states and bodily states emerges because the core of social and cognitive information processing lies in the simulation of original information [7]. There are some studies [8]-[11] about mouse movement and emotions, which all suggest a link between mouse movement and emotions. Yet, most of these studies are conducted with relatively small samples. Secondly, all these studies are dependent on the specific context of an experiment, and general link between emotions and mouse movements is not investigated. We tried to address both of previously mentioned shortcomings in our previous study [5] by using a larger sample, and avoiding specific context in our experiments. While it is hard to make a context free experiment, we were avoiding one important premise that previously mentioned studies did not. Namely, we did not use in our predictions premise that we know the target - i.e. what the user is supposed to do next. Our final goal was to make predictions about user confusion in circumstances, where the target, or information about user task is unknown. In our first study, we built a simple online computer game to collect user mouse data. We designed a game, which fills the screen with randomly arranged buttons labeled with numbers 1 to 24. All buttons were of different size and color as shown in Fig.1. Fig. 1. Game built for data collection. User has to click as fast as possible on all buttons in the right order. User task was to click on all buttons in the right order as fast as possible. As it turned out, there were some buttons in every game session that were really difficult to locate, even if they were big and literally in the front of the user’s eyes. Our goal was to identify subtasks in the game sessions, which caused a feeling of confusion for users and thereafter to identify mouse patterns associated with that state. As a ground truth about user emotions, we used retrospective self-reports that were collected right after the each game session. The user had to specify for each of the 24