IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. XX, NO. Y, MONTH YEAR 1 A Theory-Driven Approach to Predict Frustration in an ITS Ramkumar Rajendran, Sridhar Iyer, Sahana Murthy, Campbell Wilson, Judithe Sheard Abstract—The importance of affect in learning has led many Intelligent Tutoring Systems (ITS) to include learners’ affective states in their student models. The approaches used to identify affective states include human observation, self-reporting, data from physical sensors, modeling affective states, and mining students’ data in log files. Among these, data-mining and modeling affective states offers the most feasible approach in real world settings, which may involve a huge number of students. Systems using data-mining approaches to predict frustration have reported high accuracy, while systems that predict frustration by modeling affective states, not only predict a student’s affective state but also the reason for that state. In our approach we combine these approaches. We begin with the theoretical definition of frustration, and operationalize it as a linear regression model by selecting and appropriately combining features from log file data. We illustrate our approach by modeling the learners’ frustration in Mindspark, a mathematics ITS with large- scale deployment. We validate our model by independent human observation. Our approach shows comparable results to existing data-mining approaches and also the clear interpretation of the reasons for the learners’ frustration. Index Terms—Intelligent tutoring system, Affective states, Modeling Frustration, Frustration Theory. ✦ 1 I NTRODUCTION A N Intelligent Tutoring System (ITS) provides per- sonalized learning content to students based on their needs and preferences. An ITS consists of the learning content, the student model and the adaptation engine. Student models are constructed from the log files available in the ITS. The students’ interaction with ITS, such as responses to questions, number of attempts at a task, and the time taken for various activities (such as responding or reading) are captured in the ITS log file. Student models also typically contain information such as the students’ previous knowledge and background [1], from which it is possible to infer the students’ cognitive states. However, it is now well established that the learning process involves both cognitive and affective processes [2], [3], and the consideration of affective processes has been shown to achieve higher learning outcomes [4], [2]. The importance of a student’s affective component in learning has led ITS to include learners’ affective states in their student models. Baker et. al. [5] have suggested that the relevant affective states of students interacting with ITS are boredom, frustration, confusion, delight, engaged concentration and surprise. In this paper we focus on frustration. To include affective states in the student model, the • R. Rajendran is with IITB-Monash Research Academy, Indian Institute of Technology, Bombay, India and Monash University, Melbourne, Australia. E-mail: ramkumar.r@iitb.ac.in. • S. Iyer is with Department of Computer Science and Engineering and S. Murthy is with Education Technology, Indian Institute of Technology, Bombay, India. E-mail: sri, sahanamurthy@iitb.ac.in • C. Wilson and J. Sheard are with Faculty of Information Technology, Monash University, Melbourne, Australia. E-mail: campbell.wilson, Judy.Sheard@monash.edu Manuscript received on Mon xx, Year;revised Mon xx, Year. students’ affective states should be identified while they interact with the ITS. Predicting the students’ affective states, that is, attempting to determine these states while students interact with the system, is a challenging prob- lem in education research, and is the focus of several current research efforts [6], [7]. Methods that have been implemented in ITS to predict the affective state include human observation [5], [8], [9], learners’ self-reported data of their affective state [10], [11], mining the system’s log file [12], [13], modeling affective states [11], [14], face- based emotion recognition systems [4], [3], analyzing the data from physical sensors [15], [16], [10], and more recently, sensing devices such as physiological sensors [17], [18]. Advances in these methods look promising in a lab setting. However, they are not yet feasible in a large scale, real-world scenario to predict affective states [9]. The exceptions are data-mining approaches and model- ing affective states, which have additional benefits. Ex- isting systems which use data-mining approaches have reported high accuracy in predicting frustration. On the other hand, the advantage of systems which are based on modeling of affective states is that they not only predict the affective state of the learner, but also shed light on the cause for that state. In this paper, we propose an approach to identify the students’ frustration in an ITS, inspired by the better accuracy of data-mining approaches, and the use of theory in modeling affective states. We develop a model that predicts a student’s frustration while s/he interacts with an ITS. The model is derived from a theoretical definition of frustration based on the analysis of goal- blocking events, by selecting and appropriately com- bining the features in the ITS log file. The constructed features are used to form a linear regression model to predict frustration. The model helps us to understand