Technology, Mind & Society 2021 Conference Proceedings © 2021 The Author(s) http://dx.doi.org/10.1037/xxxxxxx ISBN: Curating Computer Science Educational Content with Machine Learning Analyzing Learner Ratings within an Algorithmic Recommender System Teresa M. Ober 1 , Mayank Kakodar 2 , Ying Cheng 1 , Philip Gonsalves 3 , Paul R. Brenner 1 , Emmanuel Johnson 3 , Bruno Ribeiro 2 , & Janice Zdankus 4 1 University of Notre Dame 2 Purdue University 3 YWCA Silicon Valley 4 Hewlett Packard Enterprise Machine learning (ML) can in theory be used to personalize educational content by identifying online activities aligned with learners’ interests. Yet, are learners’ self-reported ratings of activities associated with a machine learning generated recommender score? In the current study we sought to address this question using learner’s ratings of activity units (i.e., “badges”) based on a conventional 7-item Likert scale administered immediately afterwards within an online app designed to teach computer programming skills. The sample included 78 learners (MeanAge = 13.2 years, SDAge = .84 years, %female = 37.2%) enrolled in schools and after-school programs in the United States. Even after controlling for other factors, such as the position of the recommendation on a list and the number of previous selections the learner had made, as well as certain learner demographics, there was a significant positive association between overall badge approval ratings provided by learners and the recommender score. These findings provide validity evidence in support of the ML- generated recommender score by suggesting badges that the learner is likely to approve of, even when considering other relevant factors that could affect their selection of badges, such as the position of the recommendation on the list and the number of selections made by the learner. Further work should seek to establish whether the likelihood of a learner selecting a highly ranked activity is associated with improvement in more domain general attitudes towards computer programming and whether this association is moderated or robust to certain demographic factors. Keywords: computer programming, K-12 education, STEM inclusion, recommendation algorithm, machine learning, digital curation There is an ever-growing demand for a workforce with expertise in computer science and programming. In the United States, careers in computer occupations are expected to grow approximately 11% between 2019-2029, a rate nearly three times higher than the national average of 4% across all job sectors (BLS, 2021). Though there is clearly a demand for a highly skilled workforce, computer science and information technology fields generally do not reflect the general population (Code.org, CSTA, & ECEP Alliance, 2020). As such, there is great untapped potential to attract individuals from groups that are currently underrepresented in computing. Women and African American and Hispanic/Latinx individuals in the U.S. are particularly underrepresented in computing careers (NSF & NCSES, 2019). The issues of participation in the computing workforce are far reaching. However, educational interventions designed that provide positive experiences in computer science and related fields to young learners have been found to be effective in promoting conceptual understanding (Grover et al., 2015) and motivation (Papastergiou, 2009). Providing educational content in computing that aligns with learners’ background interests may be one effective means by which to trigger learners’ initial interest and initiating a cascade that results in long-term positive orientations towards the subject (Eccles, 2007). With advances in machine learning technology, it is becoming increasingly feasible to match educational content with learners' interests thus providing a fully personalized learning experience (Luan & Tsai, 2021). In the present study, we describe the design of one such educational intervention in the form of an online web application, Curated Pathways to Innovation™ (CPI™). We investigate the associations between middle and high school learners’ ratings of educational content with respect to a score produced by a machine learning algorithm designed to recommend personalized content. Developing Positive Orientations toward Careers in Computing Diverse participation in computing careers is therefore not only a societal imperative but may also be necessary for meeting the