For personal use only. Not to be reproduced without permission of The Lancet. Summary Background The inclusion of only a subset of all available evidence in a meta-analysis may introduce biases and threaten its validity; this is particularly likely if the subset of included studies differ from those not included, which may be the case for published and grey literature (unpublished studies, with limited distribution). We set out to examine whether exclusion of grey literature, compared with its inclusion in meta-analysis, provides different estimates of the effectiveness of interventions assessed in randomised trials. Methods From a random sample of 135 meta-analyses, we identified and retrieved 33 publications that included both grey and published primary studies. The 33 publications contributed 41 separate meta-analyses from several disease areas. General characteristics of the meta-analyses and associated studies and outcome data at the trial level were collected. We explored the effects of the inclusion of grey literature on the quantitative results using logistic-regression analyses. Findings 33% of the meta-analyses were found to include some form of grey literature. The grey literature, when included, accounts for between 4·5% and 75% of the studies in a meta-analysis. On average, published work, compared with grey literature, yielded significantly larger estimates of the intervention effect by 15% (ratio of odds ratios=1·15 [95% CI 1·04–1·28]). Excluding abstracts from the analysis further compounded the exaggeration (1·33 [1·10–1·60]). Interpretation The exclusion of grey literature from meta- analyses can lead to exaggerated estimates of intervention effectiveness. In general, meta-analysts should attempt to identify, retrieve, and include all reports, grey and published, that meet predefined inclusion criteria. Lancet 2000; 356: 1228–31 Introduction A meta-analysis is multifactorial. Decisions need to be made about how to handle various factors, such as language of publication, quality, and publication status, at the individual study level. Within the domain of publication status, a major factor to consider is the inclusion of grey literature (ie, studies that are unpublished, have limited distribution, and/or are not included in bibliographical retrieval system). 1 The inclusion of grey literature in a meta-analysis may help to overcome some of the problems of publication bias, and provide a more complete and objective answer to the question under consideration. However, it has been reported that only 31% of published meta-analyses include grey literature. 2 This omission may be because the nature of grey literature makes its exclusion more convenient; it is difficult to retrieve, it is frequently incomplete, and its quality may be difficult to assess. We aim to provide empirical evidence about the impact of the exclusion of grey literature from meta-analyses on the estimate of intervention effectiveness. Methods Selection of meta-analyses A sample of 135 meta-analyses were drawn randomly from an existing database of 455 meta-analyses of randomised clinical trials. 3,4 The database was established in 1996 through MEDLINE searches from 1966 to 1995 with a detailed search strategy assembled with the aid of an information specialist. Eligibility criteria To be eligible, a study has to be deemed a meta-analysis (included pooled analyses of the results of several independent primary studies), the associated studies had to be identifiable, and at least one item of grey literature (abstracts, unpublished studies, conference proceedings, graduate theses, book chapters, company reports, and applications) and one item of published work had to have been used in the generation of a summary statistic of the intervention effect. For reasons of feasibility, only meta- analyses that included binary outcomes and fewer than 100 randomised trials were considered. Data abstraction The following data were extracted via a structured form: the number of randomised trials; language of publication of these trials; year of publication of the meta-analysis and associated trials; number of patients; number and sources of grey literature; clinical area; outcome data. In meta- analyses that reported a positive outcome (such as survival) a complement outcome variable was computed (group–survivors=deaths). Outcome data (number of unwanted events and total patients in the treatment and control groups) were extracted for all independent comparisons (non- overlapping randomised trials) from the published meta- analyses or from the original trials, when necessary, by one of us, and a subset was reviewed by another. Consensus between the two was achieved for any discrepancies before data entry. ARTICLES 1228 THE LANCET • Vol 356 • October 7, 2000 Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Laura McAuley, Ba’ Pham, Peter Tugwell, David Moher Thomas C Chalmers Center for Systematic Reviews, Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Ontario, Canada (L McAuley MSc, B Pham MSc, D Moher MSc); Departments of Epidemiology and Community Medicine (P Tugwell MD, D Moher), Medicine (P Tugwell) and Paediatrics (D Moher), University of Ottawa Correspondence to: David Moher, Thomas C Chalmers Center for Systematic Reviews, Children’s Hospital of Eastern Ontario Research Institute, Room R226, 401 Smyth Road, Ottawa, Ontario K1H 8L1, Canada (e-mail: dmoher@uottawa.ca)