Screenshot Journey Auditor: A Tool to Support Analysis of Smartphone Media Consumption Journey Using Screenshot Data Chen-Chin Lin National Yang Ming Chiao Tung University Hsinchu, Taiwan spencer0929.cs04@nctu.edu.tw Jian-Hua Jiang Chen National Yang Ming Chiao Tung University Hsinchu, Taiwan novel860329@gmail.com Rebecca Ping Yu National Yang Ming Chiao Tung University Hsinchu, Taiwan rpyu@nycu.edu.tw Wan-Yun Yu National Yang Ming Chiao Tung University Hsinchu, Taiwan mwyu@nycu.edu.tw Yung-Ju Chang National Yang Ming Chiao Tung University Hsinchu, Taiwan armuro@nycu.edu.tw ABSTRACT Taking long series of screenshots for capturing and studying smart- phone users’ phone usage and media consumption has recently attracted research attention due to its advantage of capturing rich contextual information from users’ phone use journeys. However, that approach creates a high volume of screenshots that take very considerable time and efort to inspect and annotate, especially when the granularity of analysis is low: such as when distinguish- ing among media-content units (e.g, single FB posts) and detecting events in them. We therefore developed Screenshot Journey Audi- tor (SJA), a web application that identifes individual social media posts, and detects news items and other events of interest in them. It then visualizes users’ journeys ‘fow’ among these media-content units. SJA also enables researchers/coders to collaboratively correct detections online. We evaluated SJA with fve coders and received positive feedback on how the detections and visualizations made the analysis process more efcient and informative. CCS CONCEPTS · Human-centered computing → Collaborative and social computing. KEYWORDS Screenshot; Mobile phone; News consumption; Analysis tool; col- laboration ACM Reference Format: Chen-Chin Lin, Jian-Hua Jiang Chen, Rebecca Ping Yu, Wan-Yun Yu, and Yung- Ju Chang. 2022. Screenshot Journey Auditor: A Tool to Support Analysis of Smartphone Media Consumption Journey Using Screenshot Data. In Companion Computer Supported Cooperative Work and Social Computing (CSCW’22 Companion), November 8ś22, 2022, Virtual Event, Taiwan. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3500868.3559456 Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). CSCW’22 Companion, November 8ś22, 2022, Virtual Event, Taiwan © 2022 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-9190-0/22/11. https://doi.org/10.1145/3500868.3559456 1 INTRODUCTION As mobile devices become more pervasive, media consumption has gradually shifted to them from desktops [14]. Using smartphones, people may access a huge variety of media content, prominently including news in various formats [4, 6, 7, 11]. How smartphone users discover and encounter news via various channels has at- tracted considerable interest [13] from researchers using a variety of methods, including surveys [23, 24], browser-history extraction [1, 9], app-log collection [5, 6], and interviews [2, 12]. However, these methods are either retrospective (i.e., surveys, interviews) and therefore subject to recall error and reconstruction bias [10, 22], or else strictly limited in terms of how much contextual information they can capture, making interpretation difcult [20]. In addition, people have an increasing variety of channels whereby they can access and consume news on their phones, one of which comprises social-media platforms [21]. However, neither browser histories nor app logs can capture news consumption across multiple apps. Therefore, neither approach allows researchers to gain a full picture of phone users’ media consumption. To address these limitations, many studies have taken device screenshots at regular intervals to capture in-situ and cross-app behaviors [18] that can then be reconstructed [19]. For example, Hu and Lee [8] developed a desktop application, ScreenTrack, that captures its users’ software, document, and web-page use based on screenshots. Similarly, Brinberg et al. [3] collected screenshots every fve seconds and used them to analyze temporal, textual, graphical, and topical features of what appears on people’s screens, with the wider aim of describing the idiosyncratic nature of individ- uals’ daily digital lives. However, a grand challenge to this method is the amount of time and efort it takes to inspect and code series of thousands of screenshots manually, especially when the research team is interested in capturing a number of diferent events. More- over, when the granularity of these events is low (e.g., checking whether a user opens an external link in a post), inspecting nearly every screenshot is ś though necessary ś highly burdensome and tedious. Randomly sampling only a portion of collected screenshot data is a conceivable means of reducing the time and efort for data analysis, while avoiding potential systematic bias. However, adopt- ing it would mean sacrifcing data, and probably, failing to discern 110