Journal of Informetrics 8 (2014) 181–196 Contents lists available at ScienceDirect Journal of Informetrics j ourna l h o mepa ge: www.elsevier.com/locate/joi Estimating the accuracies of journal impact factor through bootstrap Kuan-Ming Chen a , Tsung-Hau Jen a,∗ , Margaret Wu b a Science Education Center, National Taiwan Normal University, No. 88, 4th Section, Ting-Chou Road, Wen-Shan District, Taipei City 11677, Taiwan, ROC b Work-based Education Research Centre (WERC), Victoria Institute for Education, Diversity and Lifelong Learning, Victoria University, PO Box 14428, Melbourne, VIC 8001, Australia a r t i c l e i n f o Article history: Received 29 May 2013 Received in revised form 18 November 2013 Accepted 20 November 2013 Keywords: Bootstrap Journal impact factor Error estimation Education Citation index a b s t r a c t The journal impact factor (JIF) reported in journal citation reports has been used to represent the influence and prestige of a journal. Whereas the consideration of the stochastic nature of a statistic is a prerequisite for statistical inference, the estimation of JIF uncertainty is necessary yet unavailable for comparing the impact among journals. Using journals in the Database of Research in Science Education (DoRISE), the current study proposes bootstrap methods to estimate the JIF variability. The paper also provides a comprehensive exposition of the sources of JIF variability. The collections of articles in the year of interest and in the preceding years both contribute to JIF variability. In addition, the variability estimate differs depending on the way a database selects its journals for inclusion. In the bootstrap process, the nested structure of articles in a journal was accounted for to ensure that each bootstrap replication reflects the actual citation characteristics of articles in the journal. In conclusion, the proposed point and interval estimates of the JIF statistic are obtained and more informative inferences on the impact of journals can be drawn. © 2013 Elsevier Ltd. All rights reserved. 1. Introduction The idea that citation frequency measures the impact of a journal dates back nearly 60 years ago (Garfield, 1955). Since 1975, the journal impact factor (JIF) reported in the Science Citation Index (SCI) has been used to represent the influence and prestige of a journal (Garfield, 1999). The 2-year and 5-year JIFs indicate the mean citation rate across articles of a journal in the two and five preceding years, respectively. The JIF obtained in this way is an average citation rate per article and allows comparison among journals regardless of the total number of articles published by a journal. However, from a statistical perspective, any measurement comes with variability or uncertainty, and this variability must be considered to make comparison or draw inferences (Greenwood, 2007; Leydesdorff, 2013; Schneider, 2013). To date, advanced statistical techniques on high-speed computers are readily accessible; therefore, the current study utilized the bootstrap method (Efron & Tibshirani, 1993) to estimate the average JIF with confidence interval (CI) and standard error (SE) for a journal. Furthermore, taking both the JIF definition and the characteristics of database into consideration, the sources underlying JIF variability are separately examined. A number of concerns about JIF has been brought about by researchers, which mainly focused on the merits and disad- vantages of using JIF (e.g., Abramo, D’Angelo, & Di Costa, 2010; Buela-Casal, Perakakis, Taylor, & Checa, 2006; Fassoulaki, ∗ Corresponding author. Tel.: +886 2 77346773; fax: +886 2 29327187. E-mail addresses: skmchen@ntnu.edu.tw (K.-M. Chen), tsunghau@ntnu.edu.tw, tsunghau.jen@gmail.com (T.-H. Jen), wu@edmeasurement.com.au (M. Wu). 1751-1577/$ – see front matter © 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.joi.2013.11.007