Data collection mode effect on feeling thermometer questions: A comparison of face-to-face and Web surveys Mingnan Liu a,⇑,1 , Yichen Wang b,2 a SurveyMonkey, 101 Lytton Avenue, Palo Alto, CA 94301, USA b NERA Economic Consulting, 4 Embarcadero Center, Suite 400, San Francisco, CA 94111, USA article info Article history: Keywords: Fact-to-face interview Web survey Feeling thermometer Data collection mode effect abstract Feeling thermometer questions are widely used in political science research to estimate people’s atti- tudes and feelings toward a political object, like a political figure or an organization. Given the popularity of the feeling thermometer question in population surveys, more work is needed to explore the measure- ment of this question type. This study examines the data collection mode effect on feeling thermometers. Using the 2012 American National Election Studies, we find that the measurement of feeling thermome- ters is not exactly comparable between face-to-face and Web surveys. Face-to-face respondents tend to provide warmer feelings, while Web respondents give relatively more reliable responses in comparison. In both survey modes, respondents are most likely to select the response options that are verbally labeled although the effect is more striking in face-to-face than Web survey. The item nonresponse between these two modes does not differ in a meaningful way. This study ends by discussing future research directions on feeling thermometer questions. Ó 2015 Elsevier Ltd. All rights reserved. 1. Introduction Feeling thermometer questions are frequently used in national surveys and political polls to measure the general population’s atti- tudes and feelings toward a political party, political figure, or some groups and organizations (Nelson, 2008). It uses a 101-point rating scale where 0 indicates very cold and unfavorable feeling while 100 indicates very warm and favorable feeling. The popularity of this type of question is primarily due to the belief that one’s feel- ings toward a certain object, such as a political party, can affect one’s subsequent behaviors, like voting (Buell & Sigelman, 1985; Greene, 1999, 2004; Kaid, Leland, & Whitney, 1992; Lauderdale, 2010; McAdams & Johannes, 1988). In the literature, there are some empirical studies that attempt to examine the measurement of 101-point feeling thermometer questions. In the study that examined the feeling thermometer questions from the 1972 Amer- ican National Election Studies (ANES), Wilcox, Sigelman, and Cook (1989) find that respondents provide systematically different ratings to the feeling thermometers based not only on their actual evaluations of various social groups but also on other factors. For example, women, non-white, and less-educated respondents tend to systematically rate all social and political groups in the survey to be warmer than men, white, and more educated respondents. In another study, Smith (1987) shows that the choice of question wording could alter the responses to feeling thermometer ques- tions. Specifically, he found that respondents rated ‘‘people on wel- fare’’ to be cooler or more unfavorable than ‘‘poor people.’’ In an attempt to identify the cause for differential stability of symbolic and nonsymbolic political attitudes, Krosnick (1991b) finds that the different measurement reliability between two different ques- tion formats may have contributed to the lower stability of non- symbolic than symbolic political attitudes. He reports that in the 1970 National Election Studies the 7-point fully verbally labeled rating scale produces higher test–retest reliability than the 101- point feeling thermometer questions. The study by Green (1988) finds that the nonrandom measurement error associated with feel- ing thermometer questions is higher than that of dichotomous like/dislike scale. Given the popularity of the feeling thermometer question in population surveys, we believe that more work is needed to explore the measurement of this question type. One of the over- looked research areas is the mode effect on feeling thermometers. The purpose of this study is to understand whether and how http://dx.doi.org/10.1016/j.chb.2015.01.057 0747-5632/Ó 2015 Elsevier Ltd. All rights reserved. ⇑ Corresponding author. Tel.: +1 765 404 6621. E-mail addresses: mingnanl@surveymonkey.com (M. Liu), yichen.wang@nera. com (Y. Wang). 1 The paper was finished when he was a Ph.D. candidate in the Michigan Program in Survey Methodology, Institute for Social Research, University of Michigan, Ann Arbor, MI, USA. 2 Tel.: +1 734 604 9366. Computers in Human Behavior 48 (2015) 212–218 Contents lists available at ScienceDirect Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh