Adjustment of meta-analyses on the basis of quality scores should be abandoned Peter Herbison a, * , Jean Hay-Smith b , William J. Gillespie c a Department of Preventive and Social Medicine, Dunedin School of Medicine, University of Otago, PO Box 913, Dunedin, New Zealand b Rehabilitation Teaching & Research Unit, Department of Medicine, Wellington School of Medicine and Health Sciences, University of Otago, New Zealand c Hull York Medical School, UK Accepted 20 March 2006 Abstract Objective: To find if a particular quality score was better than others at validly scoring the quality of randomized controlled trials, both by examining the consistency of dividing studies into high and low quality and using a large study as a reference standard. Study Design and Setting: Observational study of meta-analyses from the Cochrane Library. These had to have binary outcomes that included more than 10 studies, one or more of which randomized more than 500 people into each group. Results: Eighteen systematic reviews, with 65 meta-analyses using binary outcomes, were included and the included trials were scored for 43 different quality scores. None of these scores was better at dividing the studies in to low and high quality, and none of the scores was better over the 65 meta-analyses in making the result closer to the reference standard. Conclusion: None of the quality scores found appeared to measure quality validly. It is a mistake to assign meaning to the result of a quality score. Ó 2006 Elsevier Inc. All rights reserved. Keywords: Meta-analysis; RCT; Quality score; Bias; Systematic reviews; Adjusting for quality 1. Introduction Meta-analyses are frequently criticized for combining studies of apparently different quality without explicitly al- lowing for these differences. Differences in quality in the reporting of trials may lead to increased variation in the results, and also to bias [1,2]. This implies that the usual weighting when combining studies is not enough to adjust for differences in quality in the included studies. Adjusting a meta-analysis for quality involves changing the weight assigned to each study, so it would be expected that the outcome could change. This has been demonstrated empirically [3], although it is not always the case [4]. In spite of the knowledge that quality of the individual studies is important for the results of a meta-analysis, the proportion of meta-analyses that adjust for the perceived quality of the included studies is low [5]. Many instruments have been devised to help with the as- sessment of study quality [6]. These range in complexity from a few simple questions [7,8] to instruments that are many pages long [9,10]. Some are checklists, whose au- thors did not intend them to be summed to provide a single score. However, most instruments are scales designed to be scored and summed. Many are designed to be used in a par- ticular research area (e.g., cancer [11]), whereas others spe- cifically measure the quality of reporting, rather than the quality of the study [12]. All instruments use information from the trial report; this may not accurately represent what was done [13,14]. Detsky et al. [15] suggested four ways of using quality scores in a meta-analysis: (a) plot the quality score against the study effect, (b) use the quality score to divide the stud- ies into high and low quality, (c) use the quality score as a weight in the meta-analysis, and (d) do a cumulative meta-analysis in order of quality. All of these approaches assume that the quality score provides a valid ranking of the studies. Conflicts of interest: The authors declare that they have no conflicts of interest. * Corresponding author. Tel.: þ64-3-479-7217; fax: þ64-3-479-7298. E-mail address: peter.herbison@otago.ac.nz (P. Herbison). 0895-4356/06/$ e see front matter Ó 2006 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2006.03.008 Journal of Clinical Epidemiology 59 (2006) 1249e1256