ORIGINAL REPORTS The Effect of Standardization on the Reliability of the Philippine Board of Surgery Oral Examinations Armando C. Crisostomo, MD Department of Surgery, College of Medicine-Philippine General Hospital, University of the Philippines, Manila, Philippines BACKGROUND: Cognizant of potential problems with va- lidity and reliability, the Philippine Board of Surgery (PBS) undertook standardization of its oral examination procedures. This retrospective analytic study was conducted to determine and compare the reliability of the Philippine Board of Surgery oral examinations before and after standardization. METHODS: The records of oral examinations from June 29, 2003 to March 28, 2010 were reviewed and measures of reli- ability (intraclass correlation coefficient and kappa agreement) were computed and compared between the time periods before and after standardization and among different case content modules administered. The proportion of passers between time periods and different content areas were also compared. RESULTS: All measures of interrater reliability (intraclass cor- relation coefficient and kappa agreement) increased signifi- cantly overall and in all content areas and regardless of examinee outcome. There was also a trend to increased proportion of passers overall and significant improvement in passing rate in 4 out of 6 content areas. CONCLUSIONS: This study validates the role of standardiza- tion of the content and scoring in improving the reliability of the oral examination, which is crucial for high stakes certifying examinations. (J Surg 68:138-142. © 2011 Association of Pro- gram Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: standardization, oral examination, reliability, inter-rater reliability COMPETENCY: Medical Knowledge INTRODUCTION Despite potential problems with validity and reliability, 1,2 the oral (or “viva”) examination continues to be a commonly used assessment method in undergraduate and graduate medical ed- ucation and certification. In many countries including the Phil- ippines, nearly all medical specialty boards utilize the oral ex- amination in their certifying process. Since 1969, certification in general surgery in the Philippines has been administered by the Philippine Board of Surgery (PBS), an independent certify- ing organization. The PBS certifying examination consists of 2 parts: part 1 — a 4 hour, 240-item objective, multiple choice examination; and part 2 — a 2-hour oral examination consist- ing of 6, 20-minute examination interactions between an ex- aminee and 2 examiner raters in 6 specific content areas of general surgery (see Table 1). Cognizant of possible limitations in validity and reliability, the PBS initiated reforms in its oral examination process. Be- ginning June, 2004, standardized case scenarios were adminis- tered. In addition, standardized case-specific rating scales were utilized to assess the performance of the examinees, starting in March, 2006. This study was conducted to determine whether the stan- dardization initiatives of the PBS resulted in significant im- provements, particularly with examination reliability. METHODS The records of oral examinations conducted by the PBS were retrospectively reviewed. Included were all examinations that had 2 examiners rating the performance of the examinees. Ex- cluded were examinations that had scores of only 1 examiner. The data obtained was divided into 3 distinct time periods (see Table 2). Time Period 1 This consisted of oral examinations administered prior to the introduction of any standardization. Case scenarios were constructed independently by each examiner and utilized the “traditional” PBS grading sheet (see Fig. 1) which rated the Correspondence: Inquiries to Armando Castillo Crisostomo, MD, MHPEd, FPCS, FPSCRS, University of the Philippines College of Medicine, Department of Surgery, Taft Avenue, 1000 Manila, PH Philippines; fax: +6325230235; e-mail: armandcrs@yahoo.com Journal of Surgical Education • © 2011 Association of Program Directors in Surgery 1931-7204/$30.00 Published by Elsevier Inc. All rights reserved. doi:10.1016/j.jsurg.2010.09.013 138