Assessing technical competency during colon and rectal surgery training Jonathan Hong, MBBS, FRACS a , Sandra L. de Montbrun, MD, FRCS b , Patricia L. Roberts, MD c,d , Helen MacRae, MD, FRCS e,n a Division of General Surgery, University of Toronto, Mt Sinai Hospital and the Wilson Centre, Toronto, Ontario, Canada b Division of General Surgery, University of Toronto, St Michael's Hospital, Toronto, Ontario, Canada c Department of Colon and Rectal Surgery, Lahey Hospital and Medical Center, Burlington, MA d Tufts University School of Medicine, Boston, MA e Division of General Surgery, University of Toronto, Mt Sinai Hospital, Toronto, Ontario, Canada abstract Competence in colon and rectal surgery requires a unique set of technical skills, in addition to sound judgment and devoted patient care. Assessment of technical competence is central to defining and maintaining standards and to providing constructive trainee feedback. This review aims to appraise the instruments that are available to assess technical competence in colon and rectal surgery and discuss how these can be best utilized in assessing technical competence. & 2015 Elsevier Inc. All rights reserved. Introduction Assessment of technical competence in surgical training will become increasingly important as surgical training moves from the traditional apprenticeship model to a model of milestones, or competence-based education, and training. 1 Colon and rectal surgeons require a diverse range of skills as they operate in all quadrants of the abdomen, within the confines of pelvis, in the perianal area, and transanally. Furthermore, the subspecialty employs open, laparoscopic, endoscopic, robotic, and percutane- ous techniques. This necessitates the development of a unique surgical skill set, which, in turn demands a unique set of technical assessments. This review will focus on technical assessments germane to colon and rectal surgery, and compare these with the traditional methods of technical skill assessment. Assessment is a vital component of any training program, and should be used both formatively, to accurately identify areas that require improvement and give direction to the trainee, and summatively, to make promotion decisions and for certification standards. In a recent systematic review, technical competence was defined as a minimum standard to safely perform a procedure independently. 2 When assessing competence a single tool, or assessment, is likely to be an oversimplification, and ultimately multiple tools are required. Validity and reliability A meaningful assessment must have evidence of reliability and validity. These qualities have been extensively studied in psychol- ogy and education and subsequently applied in surgical assessment. Traditionally, validity is defined as “the extent to which a test measures what it is intended to measure” 3 and it has often been fragmented into separate types (e.g., face, content, construct, concurrent, and predictive). 4,5 However, many argue that strictly, an overarching framework of construct validity should be applied, which is based on gathering “any evidence that bears on the interpretation or meaning of the test scores.” 6 It is not a property of the test, but of the test score and its interpretation in a specific context. 7 Using this unitary framework, evidence is collected from 5 sources (Table 1) to support the interpretation of assessment results. The type and amount of evidence required by an assess- ment tool will vary depending upon the proposed use. 7 A recent systematic review suggests that the adoption of this contemporary concept of validity has been hesitant. 8 A practical guide for introducing the unified-validation approach to surgical technical skills training can be found in a 2010 review. 9 The reliability of an assessment measures its reproducibility. It can be thought of as a signal-to-noise ratio, with higher reliability tests having more “signal” and lower reliability tests having more error in measurement. The ideal test, or tests, should provide the same result for the same candidate, on different occasions, independent of the assessor, and when performing different but Contents lists available at ScienceDirect journal homepage: www.elsevier.com/locate/yscrs Seminars in Colon and Rectal Surgery http://dx.doi.org/10.1053/j.scrs.2015.04.009 1043-1489/& 2015 Elsevier Inc. All rights reserved. n Corresponding author. E-mail address: hmacrae@mtsinai.on.ca (H. MacRae). Seminars in Colon and Rectal Surgery 26 (2015) 155–159