Auditing YouTube’s Recommendation Algorithm for Misinformation Filter Bubbles IVAN SRBA, Kempelen Institute of Intelligent Technologies, Slovakia ROBERT MORO, Kempelen Institute of Intelligent Technologies, Slovakia MATUS TOMLEIN, Kempelen Institute of Intelligent Technologies, Slovakia BRANISLAV PECHER ∗ , Faculty of Information Technology, Brno University of Technology, Czechia JAKUB SIMKO, Kempelen Institute of Intelligent Technologies, Slovakia ELENA STEFANCOVA, Kempelen Institute of Intelligent Technologies, Slovakia MICHAL KOMPAN 2 , Kempelen Institute of Intelligent Technologies, Slovakia ANDREA HRCKOVA, Kempelen Institute of Intelligent Technologies, Slovakia JURAJ PODROUZEK, Kempelen Institute of Intelligent Technologies, Slovakia ADRIAN GAVORNIK, Kempelen Institute of Intelligent Technologies, Slovakia MARIA BIELIKOVA 3 , Kempelen Institute of Intelligent Technologies, Slovakia In this paper, we present results of an auditing study performed over YouTube aimed at investigating how fast a user can get into a misinformation ilter bubble, but also what it takes to “burst the bubblež, i.e., revert the bubble enclosure. We employ a sock puppet audit methodology, in which pre-programmed agents (acting as YouTube users) delve into misinformation ilter bubbles by watching misinformation promoting content. Then they try to burst the bubbles and reach more balanced recommendations by watching misinformation debunking content. We record search results, home page results, and recommendations for the watched videos. Overall, we recorded 17,405 unique videos, out of which we manually annotated 2,914 for the presence of misinformation. The labeled data was used to train a machine learning model classifying videos into three classes (promoting, debunking, neutral) with the accuracy of 0.82. We use the trained model to classify the remaining videos that would not be feasible to annotate manually. Using both the manually and automatically annotated data, we observe the misinformation bubble dynamics for a range of audited topics. Our key inding is that even though ilter bubbles do not appear in some situations, when they do, it is possible to burst them by watching misinformation debunking content (albeit it manifests diferently from topic to topic). We ∗ Also with Kempelen Institute of Intelligent Technologies. 2 Also with slovak.AI. 3 Also with slovak.AI. Authors’ addresses: Ivan Srba, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, ivan.srba@kinit.sk; Robert Moro, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, robert.moro@kinit.sk; Matus Tomlein, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, matus.tomlein@kinit.sk; Branislav Pecher, Faculty of Information Technology, Brno University of Technology, Brno, Czechia, branislav.pecher@kinit.sk; Jakub Simko, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, jakub.simko@kinit.sk; Elena Stefancova, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, elena.stefancova@kinit.sk; Michal Kompan, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, michal.kompan@kinit.sk; Andrea Hrckova, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, andrea.hrckova@kinit.sk; Juraj Podrouzek, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, juraj.podrouzek@kinit.sk; Adrian Gavornik, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, adrian.gavornik@ intern.kinit.sk; Maria Bielikova, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia, maria.bielikova@kinit.sk. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proit or commercial advantage and that copies bear this notice and the full citation on the irst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speciic permission and/or a fee. Request permissions from permissions@acm.org. © 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM. 2770-6699/2022/10-ART $15.00 https://doi.org/10.1145/3568392 ACM Trans. Recomm. Syst.