Automated survey of 8000 plan checks at eight facilities Tarek Halabi a) and Hsiao-Ming Lu Department of Radiation Oncology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts 02114 Damian A. Bernard and James C. H. Chu Department of Radiation Oncology, Rush University Medical Center, Chicago, Illinois 60612 Michael C. Kirk Department of Radiation Oncology, Mass General/North Shore Cancer Center, Danvers, Massachusetts 01923 Russell J. Hamilton Department of Radiation Oncology, University of Arizona, Tucson, Arizona 85724 Yu Lei Department of Radiation Oncology, University of Nebraska Medical Center, Omaha, Nebraska 68198 Joseph Driewer Department of Radiation Oncology, Nebraska Methodist Hospital, Omaha, Nebraska 68114 (Received 27 December 2015; revised 12 July 2016; accepted for publication 15 July 2016; published 4 August 2016) Purpose: To identify policy and system related weaknesses in treatment planning and plan check work-flows. Methods: The authors’ web deployed plan check automation solution, PlanCheck, which works with all major planning and record and verify systems (demonstrated here for  only), allows them to compute violation rates for a large number of plan checks across many facilities without requiring the manual data entry involved with incident filings. Workflows and failure modes are heavily influenced by the type of record and verify system used. Rather than tackle multiple record and verify systems at once, the authors restricted the present survey to  facilities. Violations were investigated by sending inquiries to physicists running the program. Results: Frequent violations included inadequate tracking in the record and verify system of total and prescription doses. Infrequent violations included incorrect setting of patient orientation in the record and verify system. Peaks in the distribution, over facilities, of violation frequencies pointed to suboptimal policies at some of these facilities. Correspondence with physicists often revealed incomplete knowledge of settings at their facility necessary to perform thorough plan checks. Conclusions: The survey leads to the identification of specific and important policy and sys- tem deficiencies that include: suboptimal timing of initial plan checks, lack of communica- tion or agreement on conventions surrounding prescription definitions, and lack of automation in the transfer of some parameters. C 2016 American Association of Physicists in Medicine. [http://dx.doi.org/10.1118/1.4959999] Key words: failure mode analysis, error frequencies, plan check automation, chart review, quality assurance 1. INTRODUCTION Investigators have taken several complementary approaches to better understanding treatment planning and delivery error scenarios in radiotherapy. While some efforts 1–6 resort to error reporting, others 5,7 attempt to logically identify these scenarios through formal methods such as failure mode analysis. Others still utilize logistic regression 3,8 and Bayesian network models 9 to empirically identify risk factors. Investigators have also seized on the fact that manually intensive procedures are more prone to errors. 1,10 While safety improvement measures such as failure mode analysis come at a steep cost (170 staff hours in the case of Ref. 5), automation, whether of the procedures themselves or of checks of proce- dures, comes at a negative cost in the medium to long run. 11,12 Significant safety improvements can be expected from automation. 13 For instance, reported percentages of errors directly attributable to incorrect programming of the record and verify (R&V) system range between 15% and 23%. 2,3,14 In one evaluation 4 documentation inconsistencies accounted for 42% of (more broadly defined) incidents. Though improve- ments in data transfer and in the R&V systems themselves may have reduced these errors, their effect on the percentages may be counterbalanced by increasing complexity of data transferred to and stored by these systems as well as similar improvements in other categories of errors. Regardless, these are among the types of inconsistencies easily targeted by check automation solutions. 15–19 Every effort should be made to reduce unnecessary alerts or warnings in a check automation as these dull the checker’s 4966 Med. Phys. 43 (9), September 2016 0094-2405/2016/43(9)/4966/7/$30.00 © 2016 Am. Assoc. Phys. Med. 4966