Investigating the Impact of a Real-time, Multimodal Student Engagement Analytics Technology in Authentic Classrooms Sinem Aslan 1 , Nese Alyuz 1 , Cagri Tanriover 1 , Sinem E. Mete 2 , Eda Okur 1 , Sidney K. D’Mello 3 , Asli Arslan Esme 1 1 Intel Corporation Hillsboro, OR, USA {sinem.aslan, nese.alyuz.civitci, cagri.tanriover, eda.okur, asli.arslan.esme}@intel.com 2 Bahcesehir University Istanbul, Turkey sinememine.mete@stu.bahcesehir.edu.tr 3 University of Colorado Boulder Boulder, CO, USA sidney.dmello@colorado.edu ABSTRACT We developed a real-time, multimodal Student Engagement Analytics Technology so that teachers can provide just-in-time personalized support to students who risk disengagement. To investigate the impact of the technology, we ran an exploratory semester-long study with a teacher in two classrooms. We used a multi-method approach consisting of a quasi-experimental design to evaluate the impact of the technology and a case study design to understand the environmental and social factors surrounding the classroom setting. The results show that the technology had a significant impact on the teacher’s classroom practices (i.e., increased scaffolding to the students) and student engagement (i.e., less boredom). These results suggest that the technology has the potential to support teachers’ role of being a coach in technology-mediated learning environments. CCS CONCEPTS • Applied computing -> Education -> Learning management systems • Human-centered computing -> Human computer interaction (HCI) -> Empirical studies in HCI KEYWORDS: Learning Analytics, Affective Computing, Student Engagement, Real-time, Dashboards ACM Reference format: Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E. Mete, Eda Okur, Sidney K. D’Mello and Asli Arslan Esme. 2019. Investigating the Impact of a Real-Time, Multimodal Student Engagement Analytics Technology in Authentic Classrooms. In 2019 CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland, UK. ACM, New York, NY, USA. Paper 304, 12 pages. https://doi.org/10.1145/3290605.3300534 1 INTRODUCTION Students’ increasing access to technology in classrooms brings forth new challenges for teachers. How can a teacher act as a coach, supporting student learning in technology-mediated learning environments? How can a teacher ensure that students are engaged while learning from technology? And how can a teacher identify the students in need of help and provide just-in-time personalized support to each of them? Related research shows that providing real-time learning analytics to teachers facilitates their instrumental and emotional support to students in classrooms [1, 2], which in turn improves students’ engagement, experience, and performance [3, 4]. Student engagement is an important factor for teachers to consider when personalizing learning experience [5, 6, 7, 8], as it is linked to major educational outcomes such as persistence, satisfaction, and academic achievement [6]. Towards this end, we developed a multimodal system – Student Engagement Analytics Technology (SEAT) – to detect students’ engagement-related states in real-time and provide this information to teachers so they can implement just-in- time personalized interventions. We discuss the technology and an initial evaluation study in this paper. 2 BACKGROUND AND RELATED WORK 2.1 Modeling Engagement Contemporary researchers adopt a multi-componential perspective for engagement and have operationalized person-oriented engagement in terms of affective, cognitive, and behavioral states [9]. The majority of the research investigating student engagement during learning is based on unimodal systems relying on either sensor-free data collected from student interactions and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. CHI 2019, May 4-9, 2019, Glasgow, Scotland, UK © 2019 Association for Computing Machinery. ACM ISBN 978-1-4503-5970-2/19/05…$15.00 https://doi.org/10.1145/3290605.3300534 CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK Paper 304 Page 1