A multi-faceted approach to measuring student understanding Ian T. Griffin, Kyle J. Louis, Ryan Moyer, Nicholas J. Wright, and Trevor I. Smith Department of Physics & Astronomy and Department of STEAM Education, Rowan University, 201 Mullica Hill Rd., Glassboro, NJ 08028, USA Data from the FMCE suggest that there may be inconsistencies in students’ understanding of forces when various types of motion are presented. These inconsistencies are highly evident when testing for students’ understanding using graphs. In the current study we assume that measurements of student understanding of a particular topic depend both on the student and on the instrument used to make the measurement. Multiple measurements are needed to build a more complete picture of what the student believes to be true. We compare individual student responses to 12 questions from three FMCE question clusters. Using various visualizations of the data including model analysis plots, contingency tables of student responses, and consistency plots, we identify trends that are not evident from traditional normalized-gain-based analyses. Statistical results from the χ 2 test of independence and one-way analysis of variance provide support for our findings. I. INTRODUCTION Previous results have shown that student learning on iso- morphic questions from the Force and Motion Conceptual Evaluation (FMCE) [1] depend on the details of the ques- tion setup. Smith, Wittmann, and Carter found that learning trends differed on the Force Sled question cluster (in which answer choices are presented as descriptions of forces) and the Force Graphs cluster (answers as graphs of force vs. time) [2]; however, the cause of these differences is yet unknown. Do students struggle to interpret graphs of forces as is well documented for graphs of kinematic quantities [3, 4]? How might these results inform instructional strategies? In order to begin answering these questions, we have iden- tified four questions in each of three clusters — Force Sled (FS), Force Graphs (FG), and Acceleration Graphs (AG) [5] — that present students with nearly identical descriptions of motion (velocities); we use these to define the four cases shown in Table I, each of which contains three questions that are joined by a common type of motion. We focus on indi- vidual students’ responses to questions within a given case to measure within-student coherence (does an individual give three answers that convey the same understanding?) as well as between-students consistency (do answer patterns match across multiple students?). Identifying matching responses for each of the questions in a case and interpreting each set of responses as corresponding to a particular mental model (Ta- ble II) allow us to represent and analyze our data in a variety TABLE I. Isomorphic question groups from the Force Sled (FS), Force Graphs (FG), and Acceleration Graphs (AG) question clusters. Case Described Motion Question FS FG AG 1 moving right, speeding up 1 16 22 2 moving right, steady speed 2 14 26 3 moving right, slowing down 3 18 23 4 moving left, speeding up 4 19 25 TABLE II. Definitions of models consistent with responses to the FMCE. Not all models are evident on all questions. Other models were defined that only relate to Cases 2 and 3. Model Name Model Description Cases Correct, F ∝ dv/dt Consistent with Newton’s second law: net force is proportional to the rate of change of velocity. 1–4 Common, F ∝ v Net force is proportional to velocity. 1–4 Graph as Pic- ture Graphs can be read as literal pictures of the situation. 1–4 F ∝|dv/dt| Similar to the correct model, but ignoring sign/direction. 3,4 F ∝|v|; Graph Read Left Net force is proportional to speed; reading the graph in the direction of motion. 4 of ways including model analysis [6], contingency tables [7], and consistency plots [8]. The majority of this paper is de- voted to exploring the affordances and constraints of each of these representations. We are particularly interested in doc- umenting cases in which students have a relatively high ten- dency to exist in a superposition or mixed-model state (select- ing incoherent responses within a case) [6], and identifying possible hierarchies in student learning for various questions [7]. To highlight various aspects of our results we present examples from three different schools, two of which were previously analyzed [2]. All courses used research-validated instructional materials. For brevity we present results from Cases 1 & 4; analyses of Cases 2 & 3 show similar trends. II. CONTINGENCY TABLES Both within-student coherence and between-students con- sistency for a question pair may be efficiently represented us- ing a contingency table in which each cell shows the number of students who gave a particular pair of answers to the ques- tions (Table III). We restrict our analyses to examining com- edited by Jones, Ding, and Traxler; Peer-reviewed, doi:10.1119/perc.2016.pr.028 Published by the American Association of Physics Teachers under a Creative Commons Attribution 3.0 license. Further distribution must maintain attribution to the article’s authors, title, proceedings citation, and DOI. 132 2016 PERC Proceedings,