ORIGINAL INVESTIGATIONS IN EDUCATION Simulation-based Assessment of Pediatric Anesthesia Skills James J. Fehr, M.D., F.A.A.P.,* John R. Boulet, Ph.D.,† William B. Waldrop, M.D.,‡ Rebecca Snider, M.H.S., R.N.,§ Megan Brockel, M.D., David J. Murray, M.D.# ABSTRACT Background: Assessment of pediatric anesthesia trainees is complicated by the random nature of adverse patient events and the vagaries of clinical exposure. However, assessment is critical to improve patient safety. In previous studies, a mul- tiple scenario assessment provided reliable and valid mea- sures of the abilities of anesthesia residents. The purpose of this study was to develop a set of relevant simulated pediatric perioperative scenarios and to determine their effectiveness in the assessment of anesthesia residents and pediatric anes- thesia fellows. Methods: Ten simulation scenarios were designed to reflect situations encountered in perioperative pediatric anesthesia care. Anesthesiology residents and fellows consented to par- ticipate and were debriefed after each scenario. Two pediatric anesthesiologists scored each scenario by key action checklist. The psychometric properties (reliability, validity) of the scores were studied. Results: Thirty-five anesthesiology residents and pediatric anesthesia fellows participated. The participants with greater experience administering pediatric anesthetics generally out- performed those with less experience. Score variance attrib- utable to raters was low, yielding a high interrater reliability. Conclusions: A multiple-scenario, simulation-based assess- ment of pediatric perioperative care was designed and admin- istered to residents and fellows. The scores obtained from the assessment indicated the content was relevant and that raters could reliably score the scenarios. Participants with more training achieved higher scores, but there was a wide range of ability among subjects. This method has the potential to contribute to pediatric anesthesia performance assessment, but additional measures of validity including correlations with more direct measures of clinical performance are needed to establish the utility of this approach. A NESTHESIOLOGY has pioneered the use of simula- tion technologies to enhance medical education and advance the evaluation of practitioner knowledge and skills. Simulation has been used to provide education in a number of domains in anesthesia practice that are difficult to assess. * Associate Professor, Division of Pediatric Anesthesiology, De- partments of Anesthesiology and Pediatrics, Director of Saigh Pedi- atric Simulation Center, Washington University School of Medicine, St. Louis, Missouri. † Associate Vice President for Research and Data Resources, Foundation for Advancement of International Medical Education and Research, Philadelphia, Pennsylvania. ‡ Assistant Professor, Division of Pediatric Anesthesiology, Departments of Anesthesiology and Pediatrics, Washington University School of Medicine. § Assistant Director of Education, Department of Anes- thesiology, Washington University School of Medicine.  Instructor, Division of Pediatric Anesthesiology, Department of Anesthesiol- ogy, Washington University School of Medicine. # Professor, Divi- sion of Pediatric Anesthesiology, Department of Anesthesiology, Director, Wood Simulation Center, Washington University School of Medicine. Received from the Department of Anesthesiology, Washington University School of Medicine, St. Louis, Missouri. Submitted for publication May 1, 2011. Accepted for publication September 13, 2011. Supported by the Department of Anesthesiology, Washington University School of Medicine; and the Foundation for Anesthesia Education and Research: Research in Education Grant, Rochester, Minnesota, James Fehr 2009: “Skills in Pediatric Anesthesia: A Sim- ulation-Based Assessment Program.” Salary support from the Agency for Healthcare Research and Quality, Rockville, Maryland, R01 HS018734-01 (Drs. Murray and Fehr). Address correspondence to Dr. Fehr: One Children’s Place, St. Louis Children’s Hospital, 5S31, St. Louis, Missouri 63110. fehrj@anest.wustl.edu. This article may be accessed for personal use at no charge through the Journal Web site, www.anesthesiology.org. Copyright © 2011, the American Society of Anesthesiologists, Inc. Lippincott Williams & Wilkins. Anesthesiology 2011; 115:1308 –15  This article is featured in “This Month in Anesthesiology.” Please see this issue of ANESTHESIOLOGY, page 9A. What We Already Know about This Topic • Medical simulation using multiple scenarios may enhance training and predict clinical skills in anesthesia residents, but this has not been confirmed in pediatric anesthesia What This Article Tells Us That Is New • In a study of residents and pediatric anesthesia fellows, a multiscenario simulation assessment was developed with high interrater reliability and correlation with degree of clin- ical experience Anesthesiology, V 115 • No 6 December 2011 1308 Downloaded from anesthesiology.pubs.asahq.org by guest on 06/11/2020