1 1 Probability Survey-Based Experimentation and the Balancing of Internal and External Validity Concerns 1 Paul J. Lavrakas 1 , Courtney Kennedy 2 , Edith D. de Leeuw 3 , Brady T. West 4 , Allyson L. Holbrook 5 , and Michael W. Traugott 6 1 NORC, University of Chicago, 55 East Monroe Street, Chicago, IL 60603, USA 2 Pew Research Center, Washington, DC, USA 3 Department of Methodology & Statistics, Utrecht University, Utrecht, the Netherlands 4 Survey Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI, USA 5 Departments of Public Administration and Psychology and the Survey Research Laboratory, University of Illinois at Chicago, Chicago, IL, USA 6 Center for Political Studies, Institute for Social Research, University of Michigan, Ann Arbor, MI, USA The use of experimental designs is an extremely powerful scientific methodology for directly testing casual relationships among variables. Survey researchers reading this book will find that they have much to gain from taking greater advantage of controlled experimentation with ran- dom assignment of sampled cases to different experimental conditions. Experiments that are embedded within probability-based survey samples make for a particularly valuable research method as they combine the ability to more confidently draw causal attributions based on a true experimental design that is used with the ability to generalize the results of an experiment with a known degree of confidence to the target population which the survey has sampled (cf. Fienberg and Tanur 1987, 1989, 1996). For example, starting in the late 1980s, the rapid acceptance of using technology to gather data via computer-assisted telephone interviewing (CATI), computer-assisted personal interviewing (CAPI), and computer-assisted web inter- viewing (CAWI) has made it operationally easy for researchers to embed experimental designs within their surveys. Prior to the 1990s, the history of experimentation in the social sciences, especially within psychology, essentially reflected a primary concern for maximizing the ability to identify empirically based cause-and-effect relationships – ones with strong internal validity (Camp- bell and Stanley 1966) – with little regard for whether the study could be generalized with confidence – i.e. to what extent the study had strong external validity (Campbell and Stanley 1966) – beyond the particular group of subjects/respondents that participated in the experi- ment. Tis latter concern remains highly pertinent with the current “replication crisis” facing the health, social, and behavioral sciences and its focus on “reproducibility” as a contemporary criterion of good experimentation (Wikipedia 2018a,b). Tis is not to say that prior to 1990 that all experimental social scientists were unconcerned about external validity (cf. Orwin and Boruch 1982), but rather that the research practices of most suggested that they were not. In contrast, practical experience within the field of survey 1 Te authors very much appreciate the review and valuable input received from Professor Joop Hox on an earlier version of the chapter. Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment, First Edition. Edited by Paul J. Lavrakas, Michael W. Traugott, Courtney Kennedy, Allyson L. Holbrook, Edith D. de Leeuw, and Brady T. West. © 2019 John Wiley & Sons, Inc. Published 2019 by John Wiley & Sons, Inc. Companion Website: www.wiley.com/go/Lavrakas/survey-research COPYRIGHTED MATERIAL