REVIEW What counts as validity evidence? Examples and prevalence in a systematic review of simulation- based assessment David A. Cook • Benjamin Zendejas • Stanley J. Hamstra • Rose Hatala • Ryan Brydges Received: 12 October 2012 / Accepted: 9 April 2013 Ó Springer Science+Business Media Dordrecht 2013 Abstract Ongoing transformations in health professions education underscore the need for valid and reliable assessment. The current standard for assessment validation requires evi- dence from five sources: content, response process, internal structure, relations with other variables, and consequences. However, researchers remain uncertain regarding the types of data that contribute to each evidence source. We sought to enumerate the validity evidence sources and supporting data elements for assessments using technology-enhanced simulation. We conducted a systematic literature search including MEDLINE, ERIC, and Scopus through May 2011. We included original research that evaluated the validity of simulation-based assessment scores using two or more evidence sources. Working in duplicate, we abstracted Electronic supplementary material The online version of this article (doi:10.1007/s10459-013-9458-4) contains supplementary material, which is available to authorized users. D. A. Cook Office of Education Research, Mayo Medical School, Rochester, MN, USA D. A. Cook (&) Division of General Internal Medicine, Mayo Clinic College of Medicine, Mayo 17-W, 200 First Street SW, Rochester, MN 55905, USA e-mail: cook.david33@mayo.edu B. Zendejas Department of Surgery, Mayo Clinic College of Medicine, Rochester, MN, USA e-mail: zendejas.benjamin@mayo.edu S. J. Hamstra Academy for Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada e-mail: shamstra@uottawa.ca R. Hatala Department of Medicine, University of British Columbia, Vancouver, BC, Canada e-mail: rhatala@mac.com R. Brydges Department of Medicine, University of Toronto, Toronto, ON, Canada e-mail: ryan.brydges@utoronto.ca 123 Adv in Health Sci Educ DOI 10.1007/s10459-013-9458-4