Advances in Health Sciences Education 7: 85–97, 2002. © 2002 Kluwer Academic Publishers. Printed in the Netherlands. 85 Assessing the Comparability of Standardized Patient and Physician Evaluations of Clinical Skills JOHN R. BOULET 1* , DANETTE W. MCKINLEY 1 , JOHN J. NORCINI 2 and GERALD P. WHELAN 1 1 Educational Commission for Foreign Medical Graduates, Philadelphia, PA, U.S.A.; 2 American Board of Internal Medicine, U.S.A. ( * author for correspondence; fax: +1-215-386-3309; E-mail: jboulet@ecfmg.org) Abstract. Accumulating evidence to defend decisions based on scores from evaluations is an on- going process. The purpose of this investigation was to gather additional data to support the validity of inferences made from scores on the Educational Commission for Foreign Medical Graduates’ Clinical Skills Assessment (CSA  ). This was accomplished by contrasting CSA candidate scores, and pass/fail decisions, with those obtained from the American Board of Internal Medicine’s Mini- CEX (Clinical Evaluation Exercise). Data gathering performance based on the number of unweighted history taking and physical examination checklist items adequately predicted the global ratings provided by physician observers. CSA ratings of doctor-patient communication skills correlated with mini-CEX ratings of like constructs, indicating that physician observers, using mini-CEX rating scales, are able to make realistic assessments of interpersonal skills. These results provide evidence of the convergent validity of CSA scores. Key words: clinical skills, certification tests, physician judges, standardized patients, validity Introduction The Educational Commission for Foreign Medical Graduates (ECFMG  ) is responsible for the certification of graduates of international medical schools (IMGs) who wish to enter accredited postgraduate medical training programs in the United States. Passing the Clinical Skills Assessment (CSA  ), instituted in July 1998, is one requirement for ECFMG certification. The specific purpose of CSA is to ensure that IMGs can demonstrate the ability to gather and interpret clinical patient data and communicate effectively in the English language at a level compar- able to students graduating from United States medical schools accredited by the Liaison Committee on Medical Education (LCME) (Educational Commission for Foreign Medical Graduates, 1999). The CSA is a performance-based evaluation that uses standardized patients (SPs), lay people trained to portray common clinical encounters. Extensive research on CSA indicates that valid and reliable scores can be obtained provided that the SPs are properly trained, the candidates are measured over a sufficient number and breadth of clinical encounters, and there is adherence to detailed scoring rubrics (Boulet et al., 2000; Ziv et al., 2000a, b). Nevertheless, as