Formative and Summative Analyses of Disciplinary Engagement and Learning in a Big Open Online Course Daniel T. Hickey Indiana University Bloomington IN, 47401 01-812-856-2344 dthickey@indiana.edu Joshua D. Quick Indiana University Bloomington IN, 4740 251-463-6070 jdquick@indiana.edu Xinyi Shen Beijing Normal University Beijing 100875, China 86-159-0100-1713 shenxinyi1128@gmail.com ABSTRACT Situative theories of knowing and participatory approaches to learning and assessment were used to design and then analyze learning in a “big open online course” (“BOOC”) on educational assessment. The course was delivered using Google’s Course Builder platform which was customized extensively to support both summative and formative analyses of disciplinary social engagement and individual learning. The course featured personalized “wikifolio” public assignments peer commenting, endorsement, & promotion, formal online examinations, open digital badges, and participatory learning analytics. The course was first completed by 60 students in 2013 and impressive levels of engagement and learning were documented. The course was further refined in 2014 with embedded streaming videos, embedded formative assessments, and streamlined learning analytics. Of the sixty students who registered for the course, 22 completed it. This paper illustrates the more formative learning analytics used to advance the shared discourse in the course as well as the other new features and provides detailed evidence of engagement & learning. Categories and Subject Descriptors K.3.1 [Computers in Education]: Computer Uses in Education— collaborative learning, distance learning General Terms Algorithms, Measurement, Performance, Design. Keywords Personalized learning, learning analytics, assessment, social learning analysis, analytic approaches, analytic approaches. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. LAK '15, March 16-20, 2015, Poughkeepsie, NY. Copyright is held by the owner/author(s). 1 http://en.wikipedia.org/wiki/Institute_for_Research_on_Learnin g 1. THEORETICAL ORIENTATION Theoretically speaking, this research is rooted in the situative theories of cognition that emerged from the Institute for Research on Learning in the 1990s [3, 7] In contrast to the individually- oriented learning principles from human information processing and constructivism, situative theories lead to learning principles that focus on social participation: (1) Learning is fundamentally social, (2) Knowledge is integrated in the life of communities, (3), Learning is an act of membership, (4) Knowing depends on engagement in practice, (5) Engagement is inseparable from empowerment, (6), “Failure to Learn” is the normal result of exclusion from participation, and (7) We already have a society of lifelong learners. 1 These principles formed the “metatheory” within which more specific principles were used to design and analyze an open online course on the topic of educational assessment. This work drew specific inspiration from Engle & Conant’s notion of productive disciplinary engagement [5]. This work assumes that engagement is “productive” when it leads to new questions, clarifies misunderstanding, and leads to more successful engagement by more participants; engagement is “disciplinary” when it concerns to intended topic of the course. In this course, the disciplinary knowledge consisted of the practices (e.g., guidelines for constructing various assessments) principles (e.g., reliability and validity) and policies (e.g., standardized testing, teacher evaluation) concerning assessment in schools and universities. This disciplinary knowledge was provided in the course via a widely-used and well-respected textbook on the topic, as supplemented with various online resources associated with each of the weekly assignments. In particular this work represents an effort foster interactive forms of online engagement with course content, instructors, and peers that exceedingly productive and disciplinary. The design of the course was directly shaped by Engle & Conant’s four design principles for fostering productive disciplinary engagement: (1) Problematize disciplinary content from the perspective of each learner, (2) Give students authority and position them as stakeholders and producers of disciplinary knowledge, (3) Establish disciplinary accountability and require students to