META ANALYSES SERIES Meta-analyses of small numbers of trials often agree with longer-term results Peter Herbison a, * , Jean Hay-Smith b , William J. Gillespie c a Department of Preventive and Social Medicine, University of Otago, P.O. Box 913, Dunedin 9054, New Zealand b Rehabilitation Teaching and Research Unit, Department of Medicine, University of Otago, Wellington, New Zealand c Hull York Medical School, United Kingdom Accepted 24 February 2010 Abstract Objective: Many systematic reviews include only a few studies. It is unclear whether recommendations based on these will be correct in the longer term; hence, this article explores whether meta-analyses give reliable results after only a few studies. Study Design and Setting: Cumulative meta-analysis of data from 65 meta-analyses from 18 Cochrane systematic reviews was carried out. Various measures of closeness to the pooled estimate from all trials after three and five trials were included. Changes during the ac- cumulation of evidence were noted. Results: The 95% confidence interval included the final estimate in 72% of meta-analyses after three studies and in 83% after five studies. It took a median of four (interquartile range: 1.25e6) studies to get within 10% of the final point estimate. Agreement between the results at three and five studies and the final estimate was not predicted by the number of participants, the number of events, t 2 , or I 2 . Estimates could still change substantially after many trials were included. Conclusion: Many of the conclusions drawn from systematic reviews with small numbers of included studies will be correct in the long run, but it is not possible to predict which ones. Ó 2011 Elsevier Inc. All rights reserved. Keywords: Meta-analyses; Cumulative meta-analyses; Interpretation of a meta-analysis; Validity of meta-analyses; Small number of studies; Predicting validity 1. Introduction Only a small number of trials are eligible for inclusion in most systematic reviews. A typical review in the Cochrane Library (Issue 1, 2001) contained six included trials [1]. The number of trials that contribute to each outcome con- sidered in the review will be even smaller, as some trials would not have reported all outcomes of interest. In 2004, the median number of trials in a Cochrane meta- analysis, after removing those with only one trial, was three (Joanne McKenzie, personal communication, 2005). It is not known how many trials are needed on average before a meta-analysis for an outcome in a systematic review settles down around a final value. Thus, it is unclear what weight can be put on the results of most systematic reviews that have been, and are being, done. Early trials examining a particular question may not be as well conducted as later studies and may be smaller. Some negative studies may not be published at all or take longer to get into print. The issue has been highlighted by some high-profile dis- agreements between meta-analysis of small trials and a sub- sequent very large trial [2]. In this study, we used a number of outcomes from systematic reviews in the Cochrane Library, originally col- lected for another reason, to explore how many trials are needed to get a stable answer and investigate the possible predictors of how quickly a meta-analysis settles down to a consistent estimate. 2. Methods Issue 1, 2001, of the Cochrane Library was hand searched for reviews that had binary outcomes and 10 or more trials contributing to the outcome of which one or more had more than 500 individuals per arm. These criteria were used because of another study examining the relationship of quality scores with the results of meta- analyses [3]. To establish the correct order of publication of studies, the following procedures were followed. The day, month, and year of publication were used if published by the * Corresponding author. Department of Preventive and Social Medi- cine, University of Otago, P.O. Box 913, Dunedin 9054, New Zealand. Tel.: þ64-3-479-7217; fax: þ64-3-479-7298. E-mail address: peter.herbison@otago.ac.nz (P. Herbison). 0895-4356/$ - see front matter Ó 2011 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2010.02.017 Journal of Clinical Epidemiology 64 (2011) 145e153