Evaluating the Evidence: Statistical Methods in Randomized Controlled Trials in the Urological Literature Charles D. Scales, Jr.,* Regina D. Norris,† Glenn M. Preminger,† Johannes Vieweg,† Bercedis L. Peterson,† and Philipp Dahm†,‡ From the Division of Urology, Department of Surgery (CDS, RDN, GMP), and Department of Biostatistics and Bioinformatics (BLP), Duke University Medical Center, Durham, North Carolina, and Department of Urology, College of Medicine, University of Florida, Gainesville, Florida (JV, PD) Purpose: Randomized controlled trials potentially provide the highest level of evidence to inform clinical decision making. Appropriate use of statistical methods is a critical aspect of all clinical research, including randomized controlled trials. We report the first formal evaluation to our knowledge of the statistical methods of randomized controlled trials published in the urological literature in 1996 and 2004. Materials and Methods: All human subjects randomized controlled trials published in 4 leading urology journals in 1996 and 2004 were identified for formal review. A standardized evaluation form was developed based on the Consol- idated Standards of Reporting Trials statement. Each article was evaluated by 2 independent reviewers with formal training in research design and biostatistics who were blinded to study authors and institution. Discrepancies were settled by consensus. Results: A total of 152 randomized controlled trials were reviewed (65 in 1996, 87 in 2004). The median sample size (IQR) per arm of parallel design randomized controlled trials published in 1996 and 2004 was 36 (11, 96) and 50 (26, 134) study subjects, respectively (p = 0.157). Sample size justifications were provided by 19% of studies in 1996 and 47% of studies in 2004 (p = 0.001). Of randomized controlled trials 16 (25%) vs 32 (37%) identified a single primary outcome variable (p = 0.110). Effect size estimates for primary or secondary outcome variables were provided by 5% vs 13% (p = 0.090) and the precision of the effect was detailed by 5% vs 10% of randomized controlled trials (p = 0.195). Conclusions: This formal review suggests that statistical analysis in urological randomized controlled trials has improved. However, considerable deficiencies remain. Ongoing education in applied statistics may further improve urological randomized controlled trial reporting. Key Words: statistics as topic, randomized controlled trials as topic, urology E vidence-based medicine has been defined as the “conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” 1 The central tenet of evidence-based clinical practice is the balanced integration of clinical expertise and judgment, patient and societal values, and the best available evidence. 2 The foundation for evidence- based clinical practice is clearly high quality evidence. The highest level of evidence for evaluating the efficacy of health care interventions is provided by randomized controlled trials, if well designed and executed. High qual- ity RCTs are characterized by trial design (eg randomiza- tion, blinding) as well as analytic methods (eg intent to treat analysis). Reporting of statistical analysis in RCTs is guided by the CONSORT statement which was pub- lished in 1996 and updated in 2001. 3,4 Key statistical elements identified by the CONSORT criteria include sample size calculations, intent to treat analysis, report- ing of effect size and precision, and addressing the effects of multiple analyses on trial findings. Inadequate use or reporting of these methodological safeguards has been empirically associated with bias. 5–10 Statistical hypothesis testing is the foundation of mod- ern medical research. However, statistical methods in the medical literature are often suboptimal, undermining the validity of study conclusions. 11,12 A recent assessment suggests that statistical methods in the urological litera- ture are not ideal, although RCTs comprised a small pro- portion (12%) of the designs in this investigation. 13 RCTs in the urological literature are often underpowered 14 and reporting of methodological criteria are lacking. 15,16 How- ever, no dedicated analysis of the quality of statistical methods and reporting in RCTs in the urological litera- ture has previously been published. Therefore, in a sec- ondary analysis of a previously published assessment of Submitted for publication February 11, 2008. * Financial interest and/or other relationship with Tengion, Inc. † Nothing to disclose. ‡ Correspondence: Department of Urology, University of Florida College of Medicine, Health Science Center, Box 100247, Gaines- ville, Florida 32610-0247 (telephone: 352-273-6815; FAX: 352-273- 8846; e-mail: p.dahm@urology.ufl.edu). Editor’s Note: This article is the fifth of 5 published in this issue for which category 1 CME credits can be earned. Instructions for obtaining credits are given with the questions on pages 1578 and 1579. 0022-5347/08/1804-1463/0 Vol. 180, 1463-1467, October 2008 THE JOURNAL OF UROLOGY ® Printed in U.S.A. Copyright © 2008 by AMERICAN UROLOGICAL ASSOCIATION DOI:10.1016/j.juro.2008.06.026 1463