The Practical Details of Building a CS Concept Inventory Cynthia Taylor Oberlin College Michael Clancy University of California, Berkeley Kevin C. Webb Swarthmore College Daniel Zingaro University of Toronto Mississauga Cynthia Lee Stanford University Leo Porter University of California, San Diego ABSTRACT Concept inventories (CIs) allow researchers and practitioners to measure student conceptual learning within a course or topic area. While they have enabled meaningful pedagogical change in other disciplines, there are relatively few CIs in computer science. In this paper, we report on our experiences as recent developers of a CI for basic data structures. We discuss each step along the route to a CI and offer tips based on what we have learned. We encourage others to create CIs, and we hope that this paper will serve as a practical guide through the process. CCS CONCEPTS • Social and professional topics → Computing Education. KEYWORDS concept inventory, assessment, data structures ACM Reference Format: Cynthia Taylor, Michael Clancy, Kevin C. Webb, Daniel Zingaro, Cynthia Lee, and Leo Porter. 2020. The Practical Details of Building a CS Concept Inventory. In The 51st ACM Technical Symposium on Computer Science Edu- cation (SIGCSE ’20), March 11–14, 2020, Portland, OR, USA. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3328778.3366903 1 INTRODUCTION As computer science (CS) matures as a discipline and technology becomes increasingly pervasive in society, the expectations for CS graduates have never been higher. As CS instructors, we make every effort to maximize student learning — but how can we reliably measure the effectiveness of our learning interventions? A concept inventory (CI) is an assessment designed to measure student conceptual understanding of a course. Unlike a traditional metric of student learning, such as an exam or course grade, a CI is not designed to assess individual students. Rather, a CI enables comparisons of student learning across instructors, institutions, curricula, and instructional techniques [1]. CIs empower educa- tion researchers to reliably measure the effects of interventions on student learning, and they allow practitioners to identify areas in which their students are struggling. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. SIGCSE ’20, March 11–14, 2020, Portland, OR, USA © 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-6793-6/20/03. . . $15.00 https://doi.org/10.1145/3328778.3366903 In other disciplines such as physics, CIs have inspired wide- spread change in pedagogy and facilitated large-scale studies with thousands of students [7, 13]. However, within computer science, only two validated CIs are widely available [15, 21]. We recently developed a third: a concept inventory for basic data structures (a subset of CS 2) [24] 1 . This paper describes our work building that instrument. At each step we explore the CI design space in the context of the lessons we learned. We hope that our experi- ences will concretize the CI development process and spur future CI development in other areas of CS. 2 BACKGROUND AND RELATED WORK A conception describes a belief, theory, or description previously developed to explain some behavior observed in the world. When these beliefs conflict with accepted scientific theories, they become misconceptions. One way to study student conceptions is to carefully devise a question that, answered correctly, provides a high degree of confidence in the student’s mastery. Conversely, answering in- correctly may indicate misconceptions, particularly if a student demonstrates a consistent pattern of incorrectness across a collec- tion of related questions. Such a collection of questions and their corresponding answers is referred to as a concept inventory (CI). Concept inventories emerged from the physics education com- munity, where the Force Concept Inventory (FCI) was created in response to the observation that students could successfully solve mathematical physics problems without adjusting their basic (mis)conceptions of how the world worked [16]. The FCI sparked widespread change in physics education and inspired the devel- opment of Peer Instruction [7]. The success of the FCI motivated the development of CIs in many other STEM disciplines, includ- ing chemistry [19], engineering [31], genetics [27], statistics [28], calculus [10], geoscience [20], oceanography [3], nursing [25], as- tronomy [4, 26, 35], and biology [9]. In contrast, few CIs have been developed in computer science thus far [29]. Tew and Guzdial developed the FCS1, a CI for in- troductory programming [30]. However, the FCS1 is not generally available, which led to the development of the Second CS1 Assess- ment (SCS1), an isomorphic version of the FCS1 [21]. Additionally, Herman et al. [15] developed a CI for digital logic. While other efforts explore preliminary work towards CI devel- opment for more advanced areas of CS, including algorithms [8, 11], recursion [14], advanced data structures [17], computer architec- ture [22], and operating systems [32], these efforts remain incom- plete or lack rigorous validation. Overall, the relative paucity of validated CIs for computer science, especially in advanced topics, represents both an area of great potential and a considerable need for the CS education community. 1 Available at https://groups.google.com/forum/#!forum/cs2-bdsi-concept-inventory