Development of Scenario-based Mentor Lessons An Iterative Design Process for Training at Scale ABSTRACT In this demonstration, we showcase the recent advancement of scenario-based tutor training with a focus to scale by applying the learn-by-doing approach to teaching strategies to provide socio- motivational support. These short (~15 min.) self-paced lessons use the predict-observe-explain inquiry method to develop mentor capacity in bolstering student motivation (i.e., fostering growth mindset). These custom training modules are being created to provide supplemental mentor support within the Personalized Learning 2 system, an app which combines human tutoring and student math software to improve mentoring efficiency by connecting mentors to personalized resources, such as scenario- based mentor lessons, based on individual needs. Enhancing mentor training will aid in better quality mentoring at low cost. Mentor training is most effective when scenario-based practice provides trainees with response-specific feedback. To achieve feedback at scale, we illustrate an iterative design effort toward creating selected-response tasks that maintain some of the authenticity benefits of constructed-response. These scenario-based mentor lessons will be used by national level mentoring organizations as part of our efforts to scale. CCS CONCEPTS • Applied Computing~Education • Human-Computer Interaction KEYWORDS Learning engineering, learning sciences, human-computer interaction, scenario-based learning, mentor learning This work is licensed under a Creative Commons Attribution International 4.0 License. L@S ’22, June 1–3, 2022, New York City, NY, USA © 2022 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-9158-0/22/06. https://doi.org/10.1145/3491140.3528262 ACM Reference format: Danielle R. Chine, Pallavi Chhabra, Adetunji Adeniran, Shivang Gupta, & Kenneth R. Koedinger. 2022. Development of Scenario-based Mentor Lessons: An Iterative Design Process for Training at Scale. Demonstration. In Proceedings of 2022 ACM Learning @ Scale (L@S ’22), June 1-3, 2022, New York City, NY, USA. ACM, New York, NY, USA. 3 pages. https://doi.org/10.1145/3491140.3528262 1 The Personalized Learning 2 Approach The Personalized Learning 2 (PL 2 ) system combines research-driven mentor training with AI-powered software that is designed to improve mentoring efficiency by connecting mentors to personalized resources with a click of a button. PL 2 addresses the opportunity gap among marginalized students by syncing with students’ existing math learning software and providing mentors with personalized recommendations based on mentor input and feedback and each student’s math effort and progress goals. First introduced by Schaldenbrand et al. [3] the PL 2 system consists of one web app each for both students and mentors. Students use math software that passes data to PL 2 which is used in synergy with mentor-made post-session reflections customizing student effort and progress goals. The system takes in all of these data streams and suggests resources to identify and provide solutions to address the specific challenges each student faces. The resource library is organized by mentor competency with specific strategies mentors can recommend for students or use themselves. In addition, the PL 2 team is developing scenario-based lessons which will be housed in a mentor library to improve mentor efficiency. For effective mentoring at scale, mentors must be prepared and supported by receiving quality training and consistent coaching [2]. Quality training involves having mentors practice mentoring while receiving feedback, a research-driven strategy to increase learning [5]. We have implemented scenario-based activities to provide such practice. Within such activities, having students construct open- ended responses is more authentic, but providing feedback at scale is difficult (i.e., hard to collect sufficient data to train AI for automated scoring). On the other hand, selected-response questions make it easier to scale automated feedback but may not work as Danielle R. Chine Carnegie Mellon University Pitsburgh, PA, USA dchine@andrew.cmu.edu Shivang Gupta Carnegie Mellon University Pitsburgh, PA, USA shivangg@andrew.cmu.edu Pallavi Chhabra Carnegie Mellon University Pitsburgh, PA, USA pallavic@andrew.cmu.edu Adetunji Adeniran Carnegie Mellon University Pitsburgh, PA, USA adetunja@andrew.cmu.edu Kenneth R. Koedinger Carnegie Mellon University Pitsburgh, PA, USA kk1u@andrew.cmu.edu Demos L@S ’22, June 1–3, 2022, New York City, NY, USA 469