ORIGINAL REPORTS Is the Evaluation of the Personal Statement a Reliable Component of the General Surgery Residency Application? Bobbie Ann Adair White, MA,* Mark Sadoski, PhD, † Scott Thomas, MD, ‡ and Mohsen Shabahang, PhD § *The Office of Faculty Development, Texas A&M Health Science Center, College of Medicine, Temple, Texas; † The Office of Educational Development, Texas A&M Health Science Center, College of Medicine, Bryan, Texas; ‡ Department of General Surgery, Scott & White Healthcare, Temple, Texas; § Department of General Surgery, Geisinger Medical Center, Danville, Pennsylvania BACKGROUND: Each year, fourth-year medical students spend considerable time writing and rewriting their personal statements. However, there is little evidence of what role the personal statement plays in deciding which applicants will be invited for an interview. OBJECTIVE: To evaluate the inter-rater reliability of a surgical selection committee’s ratings for both the personal statement and the application summary parts of the residency application. DESIGN: We completed a retrospective analysis of the 2007– 2008 Scott & White surgical residency application pool. From a total pool of 174 residency applications, we selected 8 (5%) applications randomly to be evaluated by 4 experienced mem- bers of the selection committee. The 4 committee members rated each personal statement on a 7-point scale, from “nega- tive—would not invite for an interview” to “positive—will in- vite for an interview.” They rated respective application sum- maries separately on a similar 7-point scale. Committee members also listed their top three reasons for assigning their scores. METHODS: Rating scores for the personal statements and the applications were analyzed for inter-rater correlation. The qual- itative data (ie, reasons for the scores) were reviewed to help the investigators profile the reasons given for very positive and very negative scores. RESULTS: For the application summaries, the correlations be- tween each pair of raters ranged from 0.79 to 0.94 with an overall average of 0.88. For the personal statements, inter-rater correlations ranged from -0.83 to 0.63 with an overall average of -0.09. CONCLUSION: These results demonstrate that the personal statements lacked objective criteria for evaluation. (J Surg 69: 340-343. © 2012 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: residency application, match, personal state- ment, application criteria COMPETENCIES: Interpersonal and Communication Skills, Professionalism, Systems Based Practice INTRODUCTION Each year, fourth-year medical students spend considerable time writing, rewriting, and seeking editorial advice about the content of their personal statements. Students believe that the quality of the personal statement may determine their chances for a residency interview. Many medical schools provide a list of “do’s” and “don’ts,” and some distribute guidelines for writing the personal statement. But the personal statement is ultimately personal, and evaluators are faced with varied styles, format, and content. It is difficult to measure objectively what makes a personal statement a “good one,” although there have been attempts in the literature to characterize it more objectively. 1 Also poorly understood is what role the personal statement plays in deciding which applicants will be invited for an interview. The National Resident Matching Program (NRMP) attempted to capture the relative contributions of application components through their 2010 NRMP Program Director Survey 2 (for additional details about their survey, see http://www.nrmp.org). Sixty-eight percent of responding directors cited the personal statement was a factor in interview selection. This percentage is higher than that of grades in required clerkships (62%), the Correspondence: Inquiries to Bobbie Ann Adair White, MA, The Office of Faculty Devel- opment, Texas A&M Health Science Center, College of Medicine, 2401 South 31st Street, Temple, TX 76508; fax: (254) 724-6810; e-mail: bawhite@medicine.tamhsc.edu Presented in part at the Association of American Medical Colleges, Joint Student Affairs and Careers in Medicine Professional Development Conference, June 11, 2011. Journal of Surgical Education • © 2012 Association of Program Directors in Surgery 1931-7204/$30.00 Published by Elsevier Inc. All rights reserved. doi:10.1016/j.jsurg.2011.12.003 340