Detecting Gaming the System in Constraint-Based Tutors Ryan S.J.d. Baker 1 , Antonija Mitrović 2 , Moffat Mathews 2 1 Department of Social Science and Policy Studies, Worcester Polytechnic Institute 100 Institute Road, Worcester MA 01609, USA rsbaker@wpi.edu 2 Intelligent Computer Tutoring Group, Computer Science and Software Engineering University of Canterbury, New Zealand {tanja.mitrovic, moffat.mathews}@canterbury.ac.nz Abstract. Recently, detectors of gaming the system have been developed for several intelligent tutoring systems where the problem-solving process is reified, and gaming consists of systematic guessing and help abuse. Constraint-based tutors differ from the tutors where gaming detectors have previously been developed on several dimensions: in particular, higher-level answers are assessed according to a larger number of finer-grained constraints, and feedback is split into levels rather than an entire help sequence being available at any time. Correspondingly, help abuse behaviors differ, including behaviors such as rapidly repeating the same answer or blank answers to elicit complete answers from the system. We use text replay labeling in combination with educational data mining methods to create a gaming detector for SQL-Tutor, a popular constraint-based tutor. This detector assesses gaming at the level of multiple- submission sequences and is accurate both at identifying gaming within submission sequences and at identifying how much each student games the system. It achieves only limited success, however, at distinguishing different types of gaming behavior from each other. Keywords: gaming the system, educational data mining, machine learning 1 Introduction In recent years, increasing attention has been paid to developing educational software which can recognize and adapt to when students game the system. A student games the system when they attempt to succeed in an educational task by systematically taking advantage of properties and regularities in the system used to complete that task, rather than by thinking through the material (cf. [4]). Detectors of gaming the system have been developed through both educational data mining/machine learning methods [5, 7, 9, 28] and through knowledge engineering approaches [1, 10, 16, 25]. These detectors have then been incorporated into interventions shown to reduce gaming and improve learning (cf. [2, 3]). However, despite the broad range of types of educational software where gaming the system has been observed (e.g. [11, 15, 23, 26, 29]), past detectors of gaming the system have been designed for fairly similar types of educational software, where students enter simple answers (a number, a word, or selecting from a set of options) with the problem-solving process reified into individual steps (though sometimes the reification only occurs after an incorrect