Patient Safety/Quality Improvement Assessments of Otolaryngology Resident Operative Experiences Using Mobile Technology: A Pilot Study Otolaryngology– Head and Neck Surgery 1–7 Ó American Academy of Otolaryngology–Head and Neck Surgery Foundation 2019 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/0194599819868165 http://otojournal.org Jenny X. Chen, MD 1,2 , Elliott Kozin, MD 1,2 , Jordan Bohnen, MD 3 , Brian George, MD 4 , Daniel G. Deschler 1,2 , Kevin Emerick, MD 1,2 , and Stacey T. Gray, MD 1,2 Sponsorships or competing interests that may be relevant to content are dis- closed at the end of this article. Abstract Objectives. Surgical education has shifted from the Halstedian model of ‘‘see one, do one, teach one’’ to a competency- based model of training. Otolaryngology residency programs can benefit from a fast and simple system to assess resi- dents’ surgical skills. In this quality initiative, we hypothesize that a novel smartphone application called System for Improving and Measuring Procedural Learning (SIMPL) could be applied in an otolaryngology residency to facilitate the assessment of resident operative experiences. Methods. The Plan Do Study Act method of quality improve- ment was used. After researching tools of surgical assess- ment and trialing SIMPL in a resident-attending pair, we piloted SIMPL across an otolaryngology residency program. Faculty and residents were trained to use SIMPL to rate resident operative performance and autonomy with a previ- ously validated Zwisch Scale. Results. Residents (n = 23) and faculty (n = 17) were trained to use SIMPL using a standardized curriculum. A total of 833 assessments were completed from December 1, 2017, to June 30, 2018. Attendings completed a median 20 assessments, and residents completed a median 14 self-assessments. All evalua- tions were resident initiated, and attendings had a 78% median response rate. Evaluations took residents a median 22 seconds to complete; 126 unique procedures were logged, representing all 14 key indicator cases for otolaryngology. Discussion. This is the first residency-wide application of a mobile platform to track the operative experiences of oto- laryngology residents. Implications for Practice. We adapted and implemented a novel assessment tool in a large otolaryngology program. Future multicenter studies will benchmark resident opera- tive experiences nationwide. Keywords autonomy, performance, training, operative independence, milestones, smartphone technology Received October 29, 2018; accepted July 17, 2019. R esidency is a critical period for the acquisition of surgical skills. Many changes have affected surgical training over the last decade, including duty hour restrictions, rising concern for patient safety, and an increas- ing emphasis on hospital productivity. There is increasing concern that, in the current educational paradigm, surgical residents may not receive the training needed to become confident and autonomous surgeons by the completion of residency. 1,2 While the otolaryngology literature is limited, surveys of general surgery residents suggest that as many as 25% of recent chief residents feel unprepared to perform certain open surgical cases independently. 3 Many otolaryngology residency programs employ the Objective Structured Assessment of Technical Skills to evalu- ate operative performance; however, this assessment can be time-consuming and cumbersome to organize for all the surgi- cal cases that residents are expected to master. 4,5 Furthermore, the ability of a detailed onetime assessment to adequately cap- ture the evolving technical skills acquired during training has not been well validated. 6-8 A longitudinal system that can track daily surgical performance assessments would provide useful and granular data to evaluate aggregate surgical 1 Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA 2 Department of Otolaryngology–Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA 3 Department of General Surgery Massachusetts General Hospital, Boston, Massachusetts, USA 4 Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan, USA This work was completed in collaboration with the Procedural Learning and Safety Collaborative. This article was presented at the AAO-HNSF 2018 Annual Meeting & OTO Experience; October 7-10, 2018; Atlanta, Georgia. Corresponding Author: Jenny X. Chen, MD, Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, 243 Charles Street, Boston, MA 02114, USA. Email: Jenny_Chen@meei.harvard.edu