Measuring Surgical Competency in Facial Trauma The Arch Bar Placement Assessment Scale Roberto L. Flores, MD,* Robert J. Havlik, MD,* Matthew Choi, MD,Þ Joseph F. Heidelman, DDS,þ Jeffrey D. Bennett, DMD,þ and Sunil Tholpady, MD, PhD* Background: Surgical education is in a period of significant change. As- sessment of surgical competency is imprecise compared with cognitive knowledge and judgment. A surgical competency measurement tool may be useful for plastic surgery training programs and certification societies. We present a validation study of a novel measurement instrument for arch bar placement and dental wire handling. Methods: An Arch Bar Placement Assessment Scale (ABPAS) was created via consensus by 2 craniofacial and 2 maxillofacial surgeons. Residents and faculty members of plastic and maxillofacial surgery (n = 20) then placed an arch bar on the lower jaw of a skull model. Performances were video recorded without revealing identities. Two study groups were created based on subjects experience level: group 1 (n = 10) previously placed fewer than 25 arch bars; group 2 (n = 10) previously placed more than 25 arch bars. Two craniofacial surgeons used the ABPAS to blindly grade surgical performance. Results: The ABPAS consisted of a 48-point rating scale that included a 23- point task-specific work list and a 25-point global rating scale. Pearson coef- ficient showed limited intraobserver (P = 0.97) and interobserver (P = 0.95) variance of test scores. The ABPAS demonstrated superior performance in group 2 in the task-specific work list [12.6 (5.5) vs 17.6 (1.5), P = 0.02], global rating scale [17.4 (4.4) vs 22 (2.1), P = 0.01], and ABPAS score [30 (9.8) and 39.6 (3.2), P = 0.01]. Conclusions: The ABPAS is a novel measurement tool which assesses tech- nical surgical skill and can identify surgical competency in arch bar placement and dental wire handling. This tool may have future use in residency training and continuing education. Key Words: facial trauma, surgical competency, competency, surgical skills, competency testing, arch bar, maxillomandibular fixation, MMF, plastic surgery, surgical testing, plastic surgery testing, plastic surgery competency (Ann Plast Surg 2014;73: 299Y303) T echnical ability is a surgical characteristic prized by both sur- geons and the lay public. Despite its obvious importance, formal assessment of technical skill falls far behind cognitive knowledge and surgical judgment by both residency training programs and board certification organizations. 1 Currently, assessment of surgical com- petency is a subjective practice. Residents are typically evaluated in an informal fashion by attending faculty who provide an assessment at the end of the service rotation or training year. This evaluation is commonly made distant from the time of performance, and the as- sessment is subjective and dependent on memory and mood. Plastic surgery training may benefit from a reliable and repro- ducible measurement tool that quantitatively assesses surgical perfor- mance. Such a measurement tool would augment subjective analysis of residents by attending staff and provide standardization for educational testing that could be followed on an institutional level. Resident per- formance could be followed over time and compared with other resi- dents of the same or different classes. In addition, the effects of any alterations in the surgical educational program could potentially be captured by such a measurement tool. Indeed, there have been recom- mendations to construct task-specific examinations to formally assess surgical competency 2 as these types of examinations have already been constructed in other surgical subspecialties. 1,3 This study introduces the Arch Bar Placement Assessment Scale (ABPAS), a novel measurement tool for surgical technical competency in facial trauma techniques. We provide evidence for validation of this tool, the purpose of which is to evaluate compe- tency in the task of arch bar placement and dental wire handling. The combination of the instructional model and the ABPAS may be useful in both the training and assessment of surgical residents. METHODS An ABPAS was created via consensus by 2 craniofacial sur- geons and 2 maxillofacial surgeons with active practices in facial trauma and was modeled after a validated surgical competency measurement tool published by Reznick et al. 1 The ABPAS consists of a 23-point task-specific work list and a 25-point global rating scale of technical performance in placing an arch bar onto mandible. A maximum score of 48 points can be awarded. The task-specific work list consists of a checklist of the essential technical steps of the op- eration (Fig. 1). A point is awarded for the proper execution of each technical step. No point is awarded if the described step is not properly executed. Care was taken to describe each step clearly and simply to leave minimal room for interpretation. The global rating scale consists of 5 elements related to surgical performance (Fig. 2). A score of 1 to 5 was awarded for each element depending on the performance with 5 being the highest score and 1 being the lowest. Institutional review board approval was obtained before performing the study. Participants (n = 20) consisted of residents and faculty of the Divisions of Plastic Surgery and Oral and Maxillofacial Surgery at Indiana University. Length of time in the residency pro- gram for trainees ranged from 1 to 4 years. Before the examination, study participants were required to complete a brief survey in which their level of training was recorded, as well as the number of arch bars previously placed and their training program. Study volunteers were divided into 2 groups based on clinical experience: group 1 (n = 10) consisted of volunteers who had previously placed fewer than 25 arch bars; group 2 (n = 10) were volunteers who had previously placed more than 25 arch bars. All testing and recording was performed at our institution’s Simulation Center, a 30,000-sq ft facility specifically created to instruct and facilitate learning while objectively measuring a trainee’s perfor- mance in a simulated health care environment. Objective Structured HEAD AND NECK SURGERY Annals of Plastic Surgery & Volume 73, Number 3, September 2014 www.annalsplasticsurgery.com 299 Received May 23, 2012, and accepted for publication, after revision, October 24, 2012. From the *Division of Plastic Surgery, Riley Hospital for Children, Indiana University Medical Center, Indianapolis, IN; †Division of Plastic Surgery, McMaster Uni- versity, Hamilton, Ontario, Canada; and ‡Division of Oral and Maxillofacial Surgery, Indiana University School of Dentistry, Indianapolis, IN. Conflicts of interest and sources of funding: none declared. Reprints: Roberto L. Flores, MD, Division of Plastic Surgery, Riley Hospital for Children, Indiana University Medical Center, 705 Riley Hospital Dr, RI 2511, Indianapolis, IN 46202. E-mail: rlflores@iupui.edu. Copyright * 2013 by Lippincott Williams & Wilkins ISSN: 0148-7043/14/7303-0299 DOI: 10.1097/SAP.0b013e31827aeb37 Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.