Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and non- randomised study designs applied in a health technology assessment setting Clare Robertson, a * † Craig Ramsay, a Tara Gurung, a Graham Mowatt, a Robert Pickard, b Pawana Sharma a and The UK Robotic Laparoscopic Prostatectomy HTA Study Group a We describe our experience of using a modified version of the Cochrane risk of bias (RoB) tool for randomised and non-randomised comparative studies. Objectives: • To assess time to complete RoB assessment • To assess inter-rater agreement • To explore the association between RoB and treatment effect size Methods: Cochrane risk of bias assessment was performed on a sample of full text primary reports included in a systematic review comparing operative techniques for radical prostatectomy. Inter-rater agreement was assessed using the kappa statistic. Results: Twenty-four studies were judged as high overall RoB, 13 were judged as low RoB and 11 were unclear. The weighted Kappa value was 0.35 indicating fair agreement. The median (range) time taken to rate each study was 30 min (10–49). The effect estimate for all studies was 0.61 (95% credible interval (CrI) 0.46–0.83) and 0.73 (95% CrI 0.29–1.75) for low risk studies. Conclusions: Although the process was time consuming, using a modified version of the RoB tool proved useful for demonstrating conservative effect estimates. That we only achieved a fair agreement between reviewers demonstrates the urgent need for further validation to improve inter-rater agreement. We suggest additional RoB levels could improve inter-rater reliability. © 2013 Crown copyright. Keywords: non-randomised; risk of bias; systematic review; Cochrane Collaboration 1. Introduction Systematic literature reviews are recognised as providing important information to health care practitioners and policy makers by summarising the best available research evidence on given health care interventions. However, inclusion of methodologically flawed studies can exaggerate treatment effects, which in turn leads to biased conclusions. Assessing risk of bias of individual included studies is therefore considered essential to ensure the validity of review conclusions (Juni et al., 2001; Wood et al., 2008). a Health Services Research Unit, University of Aberdeen, Aberdeen, UK b Institute of Cellular Medicine, Newcastle University, Newcastle upon Tyne, UK *Correspondence to: Clare Robertson, Health Services Research Unit, University of Aberdeen, 3rd Floor Health Sciences Building, Foresterhill, Aberdeen, AB25 2ZD, UK. † E-mail: c.robertson@abdn.ac.uk © 2013 Crown copyright. Res. Syn. Meth. 2014, 5 200–211 Original Article Received 21 February 2013, Revised 13 September 2013, Accepted 16 September 2013 Published online 14 November 2013 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/jrsm.1102 200