Cognitive Diagnosis Using Item Response Models Mark Wilson University of California, Berkeley, CA, USA DOI 10.1027/0044-3409.216.2.74 Abstract. In this paper, I will describe a particular approach to cognitive diagnosis that is centered on the idea of developmental assess- ment, and illustrate how data from this approach can be modeled using explanatory item response models. The developmental assessment approach starts with the idea of a progression of learning embodied in what are called progress variables. In a progress variable, student understanding is conceptualized as a continuum with successive levels of development. Effectively, these are seen as a series of student conceptions – this is the first layer of diagnosis. Then, student misconceptions are seen as particular diagnoses within the student con- ceptions, forming a second layer of diagnosis. Explanatory measurement is introduced as a way to formally model the psychometrics of this situation, using the Berkeley Evaluation and Assessment Research (BEAR) assessment system as a specific example. The discussion is illustrated with examples from student learning about selected topics in science: Earth in the Solar System, and Conceptions of Matter. The paper concludes with a discussion of further steps that match complexities in the diagnostic situation with more complex explanatory models. Keywords: developmental assessment, progress variables, item response models, BEAR assessment system (BAS) The conceptualization of cognitive diagnosis presented in this paper is based on a blending of two traditions: 1. The first is an approach to psychometric modeling, which is sometimes called developmental assessment (Masters & Forster, 1996), sometimes called construct modeling (Wilson, 2005), which centers on the idea of finding a useful (partial) ordering of how students pro- gress through a certain area of understanding. 2. The second is that part of cognitive modeling called mis- conception analysis (Confrey, 1990), which focuses at- tention on certain typical errors that students tend to make as they learn about particular content areas. Separately, each of these two approaches have made valu- able contributions to cognitive diagnosis (e.g., see Wilson & Carstensen, 2005; Wilson & Scalise, 2006; Wilson & Sloane, 2000; for the former, and Confrey, 1990; Eylon & Linn, 1988; for the latter). The general idea behind the combination of these is that each can offer some help to the other: 1. Misconception analysis offers a cognitively based inter- pretation system to construct modeling, which includes a substantive method to posit diagnoses, gather evidence for the diagnoses, and use them for professional deci- sion-making–in other words, the scores of the psycho- metric method can be given meaningful psychological interpretations. 2. Construct modeling offers the possibility of using the misconception judgments as an explicit and central part of a system for following change through an expected series of developmental levels. Note that, following the logic of Smith, diSessa, and Roschelle (1993), the misconceptions are seen in this paper in a positive way, as indicators of conceptions, in the sense of defining useful levels of the underlying psychological construct(s). A recent re-examination of the area of mis- conceptions has resulted in the development of the idea of “threshold concepts,” which are posited to be (a) transfor- mative, (b) probably irreversible, (c) integrative, (d) possi- bly bounded, and (e) troublesome (Meyer & Land, 2003), both in terms of the individual student and in terms of the educators involved, especially the teachers charged with advancing student learning across these threshold con- cepts. As an example of this combination of misconception analysis and construct modeling, consider the area of knowledge in elementary science that focuses on the “Earth in the Solar System” (to be expanded on later). Some stand- ard misconceptions that have been identified in this area of knowledge are shown in Figure 1. While each of these misunderstandings is potentially in- teresting in terms of diagnosis, it would be a struggle to give a general picture or appreciation of a student’s understanding a) It gets dark at night because the Sun goes around the Earth once a day. b) All motion in the sky is due to the Earth spinning on its axis. c) The phases of the Moon are caused by clouds covering the Moon. d) The Sun goes below the Earth at night. Figure 1. Four student statements about “Earth in the Solar System.” Zeitschrift für Psychologie / Journal of Psychology 2008; Vol. 216(2):74–88 © 2008 Hogrefe & Huber Publishers