1939-1382 (c) 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TLT.2017.2720738, IEEE Transactions on Learning Technologies 1 Predicting Students’ Disengaged Behaviors in an Online Meaning-Generation Task SungJin Nam, Gwen Frishkoff, and Kevyn Collins-Thompson Abstract—In an intelligent tutoring system (ITS), it can be useful to know when a student has disengaged from a task and might benefit from a particular intervention. However, predicting disengagement on a trial-by-trial basis is a challenging problem, particularly in complex cognitive domains. In the present work, data-driven methods were used to address two aspects of this problem: identification of predictive features at the single-trial level, and selection of accurate and robust models. Experiment data were collected in a middle-school classroom using a vocabulary training ITS. On each trial, the ITS presented a low-frequency (Tier 2 or frontier) word and prompted students to type in the word’s meaning. Single-trial measures — including the orthographic and semantic accuracy of each response, and context-sensitive measures such as interaction patterns across trials — were computed throughout the task. There were two key findings. First, as expected, different features predicted when a student was likely to be more engaged (e.g., high semantic accuracy) or less engaged (e.g., repetition of same or similar words across consecutive trials). Second, there was added value in representing context-sensitive information, which captures patterns of performance over time, as well as trial-specific information. These findings provide useful insights into effective methods for representing and modeling temporal patterns of student engagement in an ITS, especially those related to language learning. Such models may be useful in the design and implementation of adaptive tutors in complex cognitive domains like language learning. Index Terms—Disengaged behavior, vocabulary learning, prediction models, educational data mining ✦ 1 I NTRODUCTION I NTELLIGENT tutoring systems (ITS) aim to provide a student-centered environment for more effective learning. Compared to traditional learning environments, ITS can provide unique interaction opportunities between the learning system and students. For example, a typical ITS can determine an appropriate difficulty level of questions by modeling an individual student’s previous knowledge level [26], [31], or generate systematic feedback to student responses to help them develop their own learning strategies [1], [36]. By closely monitoring student behavior in ITS, educational researchers can observe students’ current progress on learning and anticipate their future performance. These advantages of ITS allow researchers to achieve deeper understanding of various behaviors of students while they interact with ITS and design more effective learning systems [14]. In order to provide a personalized learning experience, it is essential to estimate some model of each student’s state. Measuring students’ engagement level is one way to inform the ITS about the need for potential interventions. In many educational and psychological studies, engagement is considered as an important factor for predicting students’ learning outcomes [37]. In ITS, retaining student engagement is also a critical factor for ensuring the effective delivery of educational materials [44]. Previous studies have shown that engagement levels can be predicted based on various measures, such as student’s response time for individual questions [8] or reading materials [11], students’ prior domain knowledge [44], and repetitive errors or help requests [4]. Our study is part of an effort to develop a web-based contextual word learning (CWL) system that aims to help students acquire strategies for learning the meaning of an unknown word based on contextual cues in the surrounding text. This study investigates how log data from such a vocabulary-learning ITS can be used to predict specific disengaged behaviors during an online meaning-generation task. Disengaged behaviors examined in this study include students’ gaming behaviors from [4], such as systematic or repetitive incorrect attempts, and other motivation-related behaviors, like sharing responses with other students even if they were answering different practice questions from ITS. The meaning-generation task used in this study is a part of the pre-test phase of our vocabulary tutoring system. In this task, the CWL system asks free-response definition questions in which students type what they think the meaning of a new word is. Although this phase is not training oriented, it provides a well-defined yet challenging starting point for modeling disengaged behavior during a language-based task. As we show later, disengaged behaviors in this scenario are characterized by a variety of response types, such as consistent use of nonsensical or irrelevant words, names of friends or celebrities, or repetitive patterns across multiple responses (e.g., a repeated word or a sequence like “one,” “two,” “three”). In this paper, we illustrate how to extract meaningful features from the log data, including event components and textual response features, and predict disengaged labels collected from human judges. Findings in this paper will help to achieve better understanding of students’ disengaged behaviors in vocabulary-learning systems with open-ended questions, and potentially broader types of adaptive ITS in complex cognitive domains.