International Journal of Internet Science 2010, 5 (1), 20–33 ISSN 1662-5544 IJIS.NET ReCal: Intercoder Reliability Calculation as a Web Service Deen G. Freelon University of Washington, Seattle, USA Abstract: Despite the fact that intercoder reliability is an indispensable validity criterion for studies that employ content analysis, currently available options for calculating it are sharply limited both in terms of the number of coefficients they offer and the range of operating systems they support. This paper introduces ReCal, an online intercoder reliability Web service that addresses both of these limitations by offering multiple coefficients for nominal-level data and by functioning through any Web browser regardless of OS. After describing the program’s functions, its accuracy is demonstrated through a set of worked examples and a systematic comparison of its results to those of alternative reliability calculators. Finally, ReCal’s usage statistics are briefly presented. Keywords: Intercoder, interrater, interjudge, reliability, Web service, online software, Krippendorff’s alpha, nominal data Introduction Intercoder (or interrater) reliability, defined as “the extent to which independent coders evaluate a characteristic of a message or artifact and reach the same conclusion” (Lombard, Synder-Duch, & Bracken, 2002, p. 589), is an essential criterion for validating subjectively-coded data that is assessed widely throughout the social and medical sciences 1 . In mass communication research and other social science disciplines, intercoder reliability is often deployed as a component of content analysis, a standard method of summarizing the substance of a set of mediated messages. In this context, it assesses the degree to which two or more judges agree on their evaluations of a group of messages. In medicine intercoder reliability is often used to validate diagnostic and etiological classification systems for diseases and other medical conditions (Altman, 1991; Fayers & Hays, 2005; Houser & Bokovoy, 2006; Machin & Campbell, 2005); for obvious reasons, it is critical to refine these criteria so that they are as consistent as possible between doctors. Intercoder reliability is used in other scholarly fields as well, and though compiling an exhaustive list thereof would be prohibitively difficult, in general the concept applies to any research that relies upon the subjective judgments of multiple trained coders. Usage differences between scientific disciplines notwithstanding, intercoder reliability is a highly recommended (though by no means universally employed) methodological bulwark against measurement error and incoherent operational definitions in textual analysis (Hayes & Krippendorff, 2007; Lombard et al., 2002; Neuendorf, 2002). Address correspondence to Deen G. Freelon, Department of Communication, University of Washington, Seattle, USA, P.O. Box 353740, Seattle, WA 98195-3740, USA, Phone: (+1) 919 641 0442, dfreelon@u.washington.edu 1 Advanced Google Scholar searches conducted on March 25, 2010 for the term interrater reliability within the subject areas “Business, Administration, Finance, and Economics”, “Medicine, Pharmacology, and Veterinary Science”, and “Social Sciences, Arts, and Humanities” returned 19,400, 50,500, and 158,000 results respectively.