22 Framing Efect: Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases TAL AUGUST, Paul G. Allen School of Computer Science & Engineering, University of Washington, USA NIGINI OLIVIERA, Paul G. Allen School of Computer Science & Engineering, University of Washington, USA CHENHAO TAN, Department of Computer Science, University of Colorado Boulder, USA NOAH A. SMITH, Paul G. Allen School of Computer Science & Engineering, University of Washington & Allen Institute for Artifcial Intelligence, USA KATHARINA REINECKE, Paul G. Allen School of Computer Science & Engineering, University of Washington, USA Online experimentation with volunteers relies on participants’ non-fnancial motivations to complete a study, such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations to attract study participants and often use incentives, like performance comparisons, to encourage participation. Often, these study incentives are advertised using a slogan (e.g., łWhat is your thinking style?ž ). Research on framing efects suggests that advertisement slogans attract people with varying demographics and motivations. Could the slogan advertisements for studies risk attracting only specifc users? To investigate the existence of potential sample biases, we measured how diferent slogan frames afected which participants self-selected into studies. We found that slogan frames impact recruitment signifcantly; changing the slogan frame from a ‘supporting science’ frame to a ‘comparing oneself to others’ frame lead to a 9% increase in recruitment for some studies. Additionally, slogans framed as learning more about oneself attract participants signifcantly more motivated by boredom compared to other slogan frames. We discuss design implications for using frames to improve recruitment and mitigate sources of sample bias in online research with volunteers. CCS Concepts: · Human-centered computing → Empirical studies in HCI; Additional Key Words and Phrases: framing; study recruitment; volunteer-based online experimentation ACM Reference Format: Tal August, Nigini Oliviera, Chenhao Tan, Noah A. Smith, and Katharina Reinecke. 2018. Framing Efect: Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases . Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 22 (January 2018), 19 pages. https://doi.org/10.1145/3274291 Authors’ addresses: Tal August, Paul G. Allen School of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, USA, taugust@cs.washington.edu; Nigini Oliviera, Paul G. Allen School of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, USA, nigini@cs.washington.edu; Chenhao Tan, Department of Computer Science, University of Colorado Boulder, 430 UCB, Boulder, CO, 80309, USA, chenhao@chenhaot.com; Noah A. Smith, Paul G. Allen School of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, & Allen Institute for Artifcial Intelligence, USA, nasmith@cs.washington.edu; Katharina Reinecke, Paul G. Allen School of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, USA, reinecke@cs.washingont.edu. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a fee. Request permissions from permissions@acm.org. © 2018 Association for Computing Machinery. 2573-0142/2018/1-ART22 $15.00 https://doi.org/10.1145/3274291 Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 22. Publication date: January 2018.