The Logic of the Survey Experiment Reexamined Brian J. Gaines and James H. Kuklinski Department of Political Science and Institute of Government and Public Affairs, University of Illinois at Urbana-Champaign, 702 South Wright Street, Urbana, IL 61801 e-mail: bjgaines@uiuc.edu (corresponding author) e-mail: kuklinsk@ad.uiuc.edu Paul J. Quirk Department of Political Science, University of British Columbia, 2329 West Mall, Vancouver, British Columbia, Canada, V6T 1Z4 e-mail: quirk@politics.ubc.ca Scholars of political behavior increasingly embed experimental designs in opinion surveys by randomly assigning respondents alternative versions of questionnaire items. Such experi- ments have major advantages: they are simple to implement and they dodge some of the difficulties of making inferences from conventional survey data. But survey experiments are no panacea. We identify problems of inference associated with typical uses of survey experi- ments in political science and highlight a range of difficulties, some of which have straightfor- ward solutions within the survey-experimental approach and some of which can be dealt with only by exercising greater caution in interpreting findings and bringing to bear alterna- tive strategies of research. 1 Introduction Most of what we know about public opinion comes from the statistical analysis of cross- sectional survey data and, to a lesser extent, panel survey data. For over half a century, scholars have used these data to explain a wide range of phenomena, including policy preferences, economic assessments, candidate evaluations, and voting decisions, among others. The level of statistical sophistication has increased dramatically since the early days of survey research, but the basic methodological approach has changed little. Many perils attend efforts to infer causal relationships from cross-sectional survey data, as statisticians and social science methodologists continue to document. Specific Authors’ note: This paper was originally presented at the Annual Meeting of the Midwest Political Science Association, April 2004. The commentators on that panel—Darren Davis, Donald Green, and Diana Mutz—made invaluable comments. We received helpful suggestions during presentations at Columbia University, Purdue Uni- versity, and Northwestern University. We thank Thomas Rudolph for reading and commenting on an earlier version of the paper and Jamie Druckman for his encouragement from beginning to end. Psychologist Norbert Schwarz, one of the leaders of the survey experiment movement, offered invaluable insights. Robert Erikson and three anonymous reviewers gave useful advice on how to revise the original paper. Our greatest debt is to Paul Sniderman, who, more than any other single individual in political science, brought survey experiments into the mainstream. He will not agree with every argument presented here, but he has supported this project from its infancy. Ó The Author 2006. Published by Oxford University Press on behalf of the Society for Political Methodology. All rights reserved. For Permissions, please email: journals.permissions@oxfordjournals.org 1 doi:10.1093/pan/mpl008 Political Analysis Advance Access published November 16, 2006