Testing Technical Skill via an Innovative “Bench Station” Examination Richard Reznick, MD, Glenn Regehr, PhD, Helen MacRae, MD, Jenepher Martin, MD, Wendy McCulloch, RN, Toronto, Ontario, Melbourne, Australia BACKGROUND: A new approach to testing opera- tive technical skills, the Objective Structured As- sessment of Technical Skill (OSATS), formally as- sesses discrete segments of surgical tasks using bench model simulations. This study examines the interstation reliability and construct validity of a large-scale administration of the OSATS. METHODS: A 2-hour, eight-station OSATS was ad- ministered to 48 general surgery residents. Resi- dents were assessed at each station by one of 48 surgeons who evaluated the resident using two methods of scoring: task-specific checklists and global rating scales. RssuLrsJnterstation reliability was 0.78 for the checklist score, and 0.85 for the global score. Analysis of variance revealed a significant effect of training for both the checklist score, F(3,44) = 20.08, P ~0.001, and the global score, F(3,44) = 24.63, P <O.OOl . CONCLUSIONS: The OSATS demonstrates high re- liability and construct validity, suggesting that we can effectively measure residents’ technical abil- ity outside the operating room using bench model simulations. Am J Sorg. 1996;172:226-230. 0 1997 by Excerpta Medica, Inc. I f the public were surveyed as to what qualities are important in a surgeon, technical skill would un- doubtedly be near the top of the list. Yet, of all the qualities important to the development of a surgeon mea- sured in our training programs, the formal assessment of technical skill is the weakest. At present, surgical train- ing programs and certification bodies do an excellent job of assessing cognitive knowledge and a good job of as- sessing surgical judgment. There has been a heightened awareness of the need to assess the clinical skills of data gathering, data interpretation, and patient doctor com- munication, concepts that can be adequately assessed us- ing existing examination formats such as the Objective Structured Clinical Examination (OSCE).’ Professional From the Department of Surgery (RR, GR, HMac, WMc), Uni- versity of Toronto, Toronto, Ontario, Canada, and the Department of Surgery (JM), University of Melbourne, Melbourne, Australia. This work was supported by the physicians of Ontario through a grant given by the P.S.I. foundation. Requests for reprints should be addressed to Richard K. Rez- nick, MD, The Toronto Hospital, University of Toronto, EN 9-237, 200 Elizabeth Street, Toronto, Ontario, Canada M5G 2C4. Manuscript submitted August 13, 1996 and accepted in revised form October 2, 1996. qualities such as honesty, maturity, and diligence are pri- marily assessed through in-training reports, which usually represent an amalgam of preceptors informal opinions. Competence in the technical domain is, to be sure, taken seriously by all surgical training programs. However, like the domain of professional qualities, the assessment of technical skill has, for the most part, been done infor- mally and in a nonstandardized fashion. The development of a standardized test of technical skill would serve many purposes. It has the potential to set ap- propriate standards and levels of expectation for our train- ees. A standardized test would be invaluable as a feedback tool for the residents. Residents with deficits could be iden- tified early and remedial programs developed. Such a test could be used in promotion decisions, and would have the potential to validate decisions that at present are being tnade by the unratified opinions of preceptors. A test of technical skill, if reliable and valid, could allow for inter- institutional comparisons and ultimately could be used as a tool in certification and recertitication. Several years ago, the Surgical Education Research Group at the University of Toronto started a program of research with two goals. The first was to evaluate the efficacy of teaching some aspects of techtucal skill development out- side the operating room in a bench setting. The second, which we viewed as a precondition for achieving our first goal, was the development of a reliable and valid assessment tool for technical skill. Borrowing from the success of the OSCE in the domain of clinical skills, we developed a new examination, the Objective Structured Assessment of Technical Skill (OSATS). This examination is a multi- station performance based assessment of technical skill de- veloped by Martin and col1eague.s.’ The initial work on this examination was aimed at evaluating the reliability of the examination, compartng a live anitnal platform to a bench model platform, and assessing two evaluation tools, a task- specific checklist and a global rating approach. Martin et al2 reported on the results of 20 surgical residents who took the OSATS in two parallel versions: a six-station live an- imal tnodel examination and a bench model examination where the same six tasks that were tested in the animal environment were tested using bench model simulations. The reliability estimates for both examination platforms were in the moderate to high range (.66 and .74) when a global rating approach was useil to scoring, and were mixed (.61 and .33) when a task-specific checklist approach to scoring was used. There were no significant differences in the psychometric properties of the live animal version compared with the bench tnodel version. Both approaches to scoring, in both platforms of examination, were able to show increasing lev- 226 0 1997 by Excerpta Medica, Inc. 0002-9610/97f$17.00 All rights reserved. PII SOOO2-9610(97)00023-8