Data Collection Mode Effects Controlling for Sample Origins in a Panel Survey: Telephone versus Internet 1 By J. Michael Dennis 2 , Cindy Chatt 3 , Rick Li, Alicia Motta-Stanko 4 and Paul Pulliam 5 FINAL JANUARY 2005 Abstract The purpose of this research is to explore the potential for Internet panel-based survey research by conducting an experiment to investigate survey error that could hinder the validity of Internet- based survey results. In this experiment, a relatively new area of survey research - Internet surveys - is compared to a much-researched area of survey research - telephone surveys - through an experimental design that controls for sample origin. Although previous research done on telephone-Internet surveys has addressed data collection mode effects, none has been done which controls for sample origin. The present experiment is embedded in the design of the Survey of Civic Attitudes and Behaviors After 9/11, a study sponsored by RTI International and co-designed by RTII and the Odum Institute at the University of North Carolina, which is responsible for collecting 9/11 public policy-relevant attitudinal and behavioral data. Three randomly selected sample groups completed the Survey of Civic Attitudes and Behaviors After 9/11: i) an Internet survey of active Knowledge Networks (KN) panel members, ii) a telephone survey of active KN panel members, and iii) a telephone survey of persons refusing to join the KN panel and those KN panel members who did not respond to the web survey. The first two random samples were drawn from active KN panelists, but differed in the mode of data collection (Internet versus telephone). The second and third samples overlapped in terms of mode of data collection (both are telephone), but the two groups differed in terms of sample origin (active KN panel members versus refusals). The design, therefore, provides a control group of KN panelists who participated using the telephone mode of data collection. Various univariate and multivariate statistical tests were conducted in order to measure differences associated with mode of data collection and sample origins. The sources of error investigated are sample representativeness, mode effects, sample effects, panel experience effects, primacy and recency effects, the effects of visual versus aural survey administration, and non-differentiation in survey answers. Differences among sample groups were found to be due primarily to mode of data collection and panel experience, and somewhat due to sample origin. Basic differences between Internet surveys and telephone interviews could be traced back to mode of data collection. The differences found between the mode of data collection in this telephone versus Internet study were strikingly similar to the telephone versus mail mode effects found in civic attitude studies 1 The authors wish to thank RTI International for permission to analyze the data used in this paper and disseminate the findings. 2 Vice President and Managing Director, Knowledge Networks, Inc., mdennis@knowledenetworks.com 3 Research Analyst, Gallup Organization 4 Graduate student, San Jose State University, Industrial/Organizational Psychology Master’s Program 5 Senior Survey Director, RTI International, pulliam@rti.org