In recent years there has been an exponential increase in the amount of research making use of online convenience samples from crowd sourcing websites, particularly Amazon’s Mechanical Turk. While several studies have explored the features of MTurk’s demographics there have been few comparable studies for other crowdsourcing platforms outside of the US. To help address this we conducted the first large scale study of Lancers.co.jp, Japan’s largest crowdsourcing website, and assessed its value as a participant pool for social science research, by addressing 3 Core Questions: 1. What are the common demographics of Lancers’ users? 2. Do the responses of Lancers users meet acceptable standards of quality? 3. What do users consider to be fair rates of compensation? To collect our sample we posted a task to Lancers that required users to complete a quick survey (avg. 20 mins) for a small amount of compensation (avg. 500 JPY). To insure a broad representation of users we released the tasks in 10 staggered batches over 14 days (including weekends) in morning, afternoon, and evening slots. After excluding repeat participants and incomplete surveys (N=86) our final sample size was N=1000 (M=45%, F=55%, 99.8% Japanese, 98.5% East Asian Ethnicity). Lancer Demographics A critical survey of users of a popular Japanese crowdsourcing website Christopher KAVANAGH (University of Oxford), Robert THOMSON & Masaki YUKI (Hokkaido University) Email contact: christopher.kavanagh@anthro.ox.ac.uk Introduction Conclusions Demographics Methodology & Sample Quality of Responses Lancer Usage & Fair Compensation % Familiar with Behavioral Task Dictator Game Ultimatum Game Amazon MTurk Users (Chandler et al. 2013) 40% 56% Lancer Users 5% 5% • All 47 prefectures represented in sample • Higher responses from expected urban population hubs Geographic Diversity Lancer users are less familiar with common experimental tasks than MTurk users Age Range • Broad age range 18-73, M=35 (SD=9.5) 0% 5% 10% 15% 20% 25% 18-25 26-30 31-35 36-40 41-50 51-60 60+ • Overall even split between single and w/partners • But important gender differences: Males mostly single (68%) and Females mostly in relationships or married (71%) Single 43% In a relationship/married 50% Other 7% Relationship Status Occupation 8.00% 6.20% 11.30% 12.60% 17.20% 20.60% 24.10% 0% 5% 10% 15% 20% 25% 30% Other Student Unemployed Self Employed Part Time Worker House Wife/Husband Company Employee US MTurk (Smith et al. 2016) Non-US MTurk (Smith et al. 2016) Lancers (Current Study) % who Passed Attention Check 83.2% 58.5% 82.8% Most Lancer users passed an end of survey attention filter similar to US MTurk users Attention check used in Lancer study: Q. Which site did you access this survey from? (This is a question to assess your attention so please select Amazon MTurk as your answer) Lancer users’ responses are consistent with large nationally representative samples End of Year Survey (2014) Lancers (Current Study) Happy with Life? Very Happy- 10%, Happy- 48%, Neutral- 30%, Unhappy- 3%, Very Unhappy- 5% Very Happy- 9%, Happy- 43%, Neutral- 30%, Unhappy- 13%, Very Unhappy- 4% Fight for Country? Yes- 10%, No- 43% Yes- 15%, No-45% What’s Important? Nationality- 18%, Local Area- 14%, Ethnicity- 3%, Religion- 3% Nationality- 27%, Local Area- 17%, Ethnicity- 2%, Religion- 3% Do you Trust? Politicians- 3%, Journalists- 11%, Judges- 45%, Police- 34% Politicians- 4%, Journalists- 14%, Judges- 53%, Police- 38% World Values Survey (2010) Lancers (Current Study) Think of Purpose? Often- 21%, Sometimes- 59%, Rarely-14% Often-33%, Sometimes- 56%, Rarely- 10% Importance- Friends Very- 45%, Rather- 47%, Not Very- 5% Very- 27%, Rather- 50%, Not Very- 18% Importance- Work Very- 52%, Rather- 32%, Not Very- 8% Very- 27%, Rather- 50%, Not Very- 20% 1 2 3 Validated psychology scales displayed expected psychometric properties 4 Expected Factor Loadings Scale Reliability (Lancers) Scale Reliability (Other Japan Sample) Rosenberg Self-Esteem Scale ✓ α=.90 α=.81 (Mimura & Griffiths 2007) Identity Fusion Scale ✓ α=.88 α=.90 (Kavanagh et al., forthcoming) Relational Mobility Scale ✓ α=.84 α=.76 (Schug et al. 2010) Supernatural Belief Scale ✓ α=.89 α=.88 (Kavanagh & Jong, forthcoming) 51.9% 21.3% 15.0% 5.1% 0% 10% 20% 30% 40% 50% 60% 0-5 hr 6-10 hrs 11-20 hrs 21-30 hrs Avg. Hours worked per week • 50% of users are working less than 5hrs pw • Average hours pw: M=9.6 (SD=11.5) • 11% are heavy users doing 20+ hours 1. Lancers demographics indicate diverse regional representation, proportional gender balance, and an almost entirely Japanese user base. 2. There are important gender differences in social demographics with most males being single, company employees and most females being married, house wives. 3. Multiple indicators suggest that the quality of responses is comparable to standards of typical academic samples 4. Despite income being the primary motivation most users feel that payments are unsatisfactory. It is recommended that researchers pay at least 400 JPY per 30 min task. 8.2% 19.0% 42.9% 10.5% 16.4% 1.9% 0.4% 0% 10% 20% 30% 40% 50% Extremely Unsatisfied Moderately Unsatisfied A little Unsatisfied Neutral A little satisfied Moderately Satisfied Extremely Satisfied Most users are unsatisfied with compensation they receive References Chandler, J., Mueller, P., & Paolacci, G. (2014) Non-naïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral. Behavior Research Methods, 46 (1), 112-130. Smith, S. M., Roster, C. A., Golden, L. L., & Albaum, G. S. (2016). A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples. Journal of Business Research, 69(8), 3139-3148. Kavanagh, C., Tsuchida, S., Jong, J., Yuki, M. & Whitehouse, H. (forthcoming) Ritual Pain & Social Gain: Examining the impact of collective dysphoric arousal on in-group preference and cooperation, using a novel artificial ritual paradigm. Kavanagh, C. & Jong. J. (forthcoming) Contemporary beliefs and practices related to religion and the supernatural in Japan. Mimura, C., & Griffiths, P. (2007). A Japanese version of the Rosenberg Self-Esteem Scale: Translation and equivalence assessment. Journal of Psychosomatic Research, 62(5), 589-594. Schug, J., Yuki, M., & Maddux, W. W. (2010). Relational Mobility Explains Between-and Within-Culture Differences in Self-Disclosure to Close Friends. Psychological Science, 21(10), 1471-1478. WinGallup, 2014. End of Year Survey 2014- Japan Country Data. World Values Survey Association (2014) World Values Survey Wave 6- Japan Country Data. Desired Fair Payment for 30 minute task Actual Payment for 30 minute task Payment in JPY 400 - Median 396 - Mean 100 - Median 191 - Mean 300 JPY Difference! Type of Task Proportion of Tasks (%) Writing Tasks 38% Marketing Questionnaire 22% Visiting/Testing Websites 17% Academic Questionnaire 7% Other (Data Entry, Search Results) 14% Currently academic studies are relatively rare on Lancers • 77% of users use Lancers for extra income, 11% as primary income 1 2 3