Applying Kane’s validity framework to a simulation based assessment of clinical competence Walter Tavares 1,2,3 • Ryan Brydges 1 • Paul Myre 5 • Jason Prpic 5 • Linda Turner 6 • Richard Yelle 4 • Maud Huiskamp 6 Received: 22 February 2017 / Accepted: 22 October 2017 Ó Springer Science+Business Media B.V. 2017 Abstract Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particu- larly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane’s validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assess- ment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assess- ment data. Data evaluating ‘‘Scoring’’ included: (a) desirable tool characteristics, with acceptable inter-item correlations (b) strong item-total correlations (c) low error variance for items and raters, and (d) strong inter-rater reliability. Data evaluating ‘‘Generaliz- ability’’ included: (a) a robust sampling strategy capturing the majority of relevant medical Electronic supplementary material The online version of this article (http://doi.org/10.1007/s10459-017- 9800-3) contains supplementary material, which is available to authorized users. & Walter Tavares walter.tavares@utoronto.ca 1 The Wilson Centre, Department of Medicine, University of Toronto/University Health Network, 200 Elizabeth Street, 1ES-565, Toronto, ON M5G 2C4, Canada 2 Post-MD Education (Post-Graduate Medical Education/Continued Professional Development), University of Toronto, Toronto, ON, Canada 3 Paramedic and Senior Services, Community and Health Services Department, Regional Municipality of York, Newmarket, ON, Canada 4 Ornge Transport Medicine, Base Hospital and Clinical Affairs, Mississauga, ON, Canada 5 Health Sciences North Base Hospital, Sudbury, ON, Canada 6 Sunnybrook Base Hospital, Toronto, ON, Canada 123 Adv in Health Sci Educ DOI 10.1007/s10459-017-9800-3