REVIEW ARTICLES Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data Eric I. Benchimol a,b,c,d,e,f,g, * , Douglas G. Manuel a,h,i,j,k , Teresa To a,c,d , Anne M. Griffiths b,e , Linda Rabeneck a,d,l , Astrid Guttmann a,d,e,m a The Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada b Division of Gastroenterology, Hepatology and Nutrition, The Hospital for Sick Children, Toronto, Ontario, Canada c Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, Ontario, Canada d Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada e Department of Paediatrics, University of Toronto, Toronto, Ontario, Canada f Division of Gastroenterology, Hepatology and Nutrition, Children’s Hospital of Eastern Ontario, Ottawa, Ontario, Canada g Department of Pediatrics, University of Ottawa, Ottawa, Ontario, Canada h Department of Family Medicine, University of Ottawa, Ottawa, Ontario, Canada i Department of Epidemiology and Community Medicine, University of Ottawa, Ontario, Ottawa, Canada j Ottawa Hospital Research Institute, Ottawa, Ontario, Canada k Statistics Canada, Ottawa, Ontario, Canada l Department of Medicine, University of Toronto, Toronto, Ontario, Canada m Division of Paediatric Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada Accepted 7 October 2010 Abstract Background and Objectives: Validation of health administrative data for identifying patients with different health states (diseases and conditions) is a research priority, but no guidelines existfor ensuring quality. We created reporting guidelines for studies validating admin- istrative data identification algorithms and used them to assess the quality of reporting of validation studies in the literature. Methods: Using Standards for Reporting of Diagnostic accuracy (STARD) criteria as a guide, we created a 40-item checklist of items with which identification accuracy studies should be reported. A systematic review identified studies that validated identification algorithms using administrative data. We used the checklist to assess the quality of reporting. Results: In 271 included articles, goals and data sources were well reported but few reported four or more statistical estimates of ac- curacy (36.9%). In 65.9% of studies reporting positive predictive value (PPV)/negative predictive value (NPV), the prevalence of disease in the validation cohort was higher than in the administrative data, potentially falsely elevating predictive values. Subgroup accuracy (53.1%) and 95% confidence intervals for accuracy measures (35.8%) were also underreported. Conclusions: The quality of studies validating health states in the administrative data varies, with significant deficits in reporting of markers of diagnostic accuracy, including the appropriate estimation of PPVand NPV. These omissions could lead to misclassification bias and incorrect estimation of incidence and health services utilization rates. Use of a reporting checklist, such as the one created for this study by modifying the STARD criteria, could improve the quality of reporting of validation studies, allowing for accurate application of algo- rithms, and interpretation of research using health administrative data. Ó 2011 Elsevier Inc. All rights reserved. Keywords: Health administrative data; Misclassification bias; Diagnostic accuracy; Sensitivity and specificity; Predictive values; Health services research; Epidemiology 1. Introduction Health services and epidemiologic research are best con- ducted with population-level data. This helps ensure the ap- propriate estimation of incidence and prevalence rates, the minimization of referral bias, and the overall generalizabil- ity of the study conclusions to the population of interest. Because prospective clinical registries and retrospective This research was conducted with the support of a Clinical Research Award from the American College of Gastroenterology. * Corresponding author. Division of Gastroenterology, Hepatology and Nutrition, Children’s Hospital of Eastern Ontario, 401 Smyth Road, Ottawa, Ontario K1H 8L1, Canada. Tel.: þ1-613-737-7600; fax: þ1-613- 738-4854. E-mail address: ebenchimol@cheo.on.ca (E.I. Benchimol). 0895-4356/$ - see front matter Ó 2011 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2010.10.006 Journal of Clinical Epidemiology 64 (2011) 821e829