How Effective are Intelligent Tutoring Systems in Computer Science Education? John C. Nesbit Faculty of Education Simon Fraser University Burnaby, Canada nesbit@sfu.ca Olusola O. Adesope Faculty of Education Washington State University Pullman, U.S.A olusola.adesope@wsu.edu Qing Liu 1 , Wenting Ma 2 Faculty of Education Simon Fraser University Burnaby, Canada 1 qla22@sfu.ca 2 wentingm@sfu.ca Abstract—A meta-analysis on the effectiveness of Intelligent Tutoring Systems (ITS) in computer science education compared the learning outcomes of ITS and non-ITS instruction. A search of the literature found 22 effect sizes (involving 1,447 participants) that met the pre-defined inclusion criteria. Although most of the ITS were used to teach programming, other topics such as database design and computer literacy were also represented. There was a significant overall effect size favoring the use of ITS. There was a significant advantage of ITS over teacher-led classroom instruction and non-ITS computer-based instruction. ITS were more effective than the instructional methods to which they were compared regardless of whether they modeled misconceptions and regardless of whether they were the primary means of instruction or were an integrated component of learning activities that included other means of instruction. Keywords-tutor; student model; adaptive instruction; effect size; constraint-based; Bayesian network I. INTELLIGENT TUTORING SYSTEMS IN COMPUTER SCIENCE EDUCATION An Intelligent Tutoring System (ITS) is a type of computer-assisted learning software that models the cognitive and emotional states of individual learners with the goal of adapting and personalizing instruction. Since the earliest attempts at creating ITS in the 1970s, they have been used to teach a variety of subjects including many areas of science, mathematics, the humanities, and the social sciences. Understandably, topics within computer science are among the most common subject domains for ITS research. ITS have been developed to teach programming (C, C++, Java, Lisp, QBASIC), linked lists, software design, SQL query formulation, database design, system security, computer literacy and general introductory computer science. ITS have been used in computer science education to provide detection and remediation of misconceptions [1], immediate diagnostic feedback and recognition of cognitive mastery in programming [2], automated guidance as students engage with especially difficult aspects of programming [3], peer agents that act as partners in pair programming [4], and natural language discussion of topics such as program planning [5], hardware, operating systems, and the internet [6]. A. Are Intelligent Tutoring Systems Effective? A meta-analysis is a type of review that aggregates the results from an area of research by calculating weighted mean effect sizes. Recent meta-analyses of studies prior to 2012 that compared the learning outcomes of ITS to alternate modes of instruction found that, aggregating across all subject areas, effect sizes ranged from .35 to .41 standard deviations in favor of the ITS condition [7; 8]. The effect size of ITS instruction depends on the type of instruction it is compared with. ITS were found to outperform large-group instruction (.42 SD), non-ITS computer-based instruction (.57 SD), and text or workbooks (.35 SD) but produce lower achievement than one-to-one human tutoring (-.11 SD) [8]. The research reported here is a meta-analysis of the effects of using ITS for computer science education. As our recent meta-analysis covering all subject areas [8] found that ITS teaching computer science topics produced an effect size of approximately 0.5 SD, the purpose of our research is to break down the effects of ITS in computer science according to several key moderator variables such as type of comparison treatment, type of student modeling, whether the ITS provided feedback, and whether the ITS modeled misconceptions. A secondary purpose of this research is to survey the features of existing research evaluating ITS in computer science education and identify areas and topics in need of further evaluative research. B. What Exactly is an Intelligent Tutoring System? The meta-analysis required a precise definition of ITS so we could objectively decide which studies to include and exclude. The definition we constructed is consistent with expert usage and understanding of ITS [9]. An intelligent tutoring system is software that interacts with a student to promote learning and: a) performs one or more tutoring functions such as asking questions, assigning tasks, offering hints, providing feedback, or answering questions; b) uses student input to model the student’s cognitive, motivational or emotional states in a multidimensional space; c) applies the modeling functions in (b) to adapt one or more tutoring functions in (a). Student modeling in ITS may take the form of a persistent student model, often represented as an ‘overlay’ of 2014 IEEE 14th International Conference on Advanced Learning Technologies 978-1-4799-4038-7/14 $31.00 © 2014 IEEE DOI 10.1109/ICALT.2014.38 99