312 | wileyonlinelibrary.com/journal/ssm School Science and Mathematics. 2019;119:312–326. © 2019 School Science and Mathematics Association 1 | INTRODUCTION The improvement of validation methods for educational as- sessments with the goal of insuring inferences are appropri- ate and sound (American Educational Research Association [AERA], American Psychological Association [APA], and National Council on Measurement in Education [NCME], 2014) has become a mantra for educators in recent years. With regard to the assessment of 21st Century Skills specifically, concerns have arisen that measures used to assess these skills were not rigorously developed (Lai & Viering’s, 2012). While there are numerous methods in which the outcomes on ed- ucational assessments may be validated, investigating multi- ple forms of validity and reliability evidence through various methodological approaches is considered best practice. This study aims to expand the literature on developing and vali- dating educational assessments through detailing the valida- tion process used for the development of a 21st Century Skills global rating scale. Specifically, we examine whether validity and reliability evidence support implementation of the 21st Century Skills Assessment (Johnson & Sondergeld, 2016). 2 | REVIEW OF LITERATURE 2.1 | Importance of 21st Century Skills As we enter the “innovation age,” the ability to use creativity, problem‐solving, and critical thinking skills in tandem with content knowledge to solve daily and global problems is of utmost importance (i.e., Aldrin, Johnson, Aldrin, & Saikia, in press; Johnson, 2018; Modi, 2011). Industry and business stakeholders have grown increasingly concerned about the skills gap of potential candidates and current workforce, as organizations seek employees who can collaboratively think and do (e.g., Walton & Johnson, 2015). Research suggests that 21st Century Skills are essential for a STEM‐literate so- ciety and for a wide range of careers; this includes jobs re- quiring less than a high school diploma and ranges upward to high‐wage professional‐oriented positions requiring special training and/or higher education degrees (e.g., Bybee, 2013; Johnson, Peters‐Burton, & Moore, 2015; Levy & Murnane, 2004). In response to these concerns, the Partnership for 21st Century Learning developed a guiding document to be Received: 30 July 2019 | Revised: 30 July 2019 | Accepted: 30 July 2019 DOI: 10.1111/ssm.12355 RESEARCH PAPER – INTEGRATED STEM EDUCATION Development and validation of a 21st Century Skills Assessment: Using an iterative multimethod approach Toni A. Sondergeld 1 | Carla C. Johnson 2 1 School of Education, Drexel University, Philadelphia, Pennsylvania 2 College of Education, North Carolina State University, Raleigh, North Carolina Correspondence Toni A. Sondergeld, PhD, School of Education, Drexel University, 2401 Market St., Philadelphia, PA 19104. Email: tas365@drexel.edu Abstract In response to the call for more rigorously validated educational assessments, this study used an iterative multimethod validation process to develop and validate out- comes from the 21st Century Skills Assessment global rating scale. Qualitative and quantitative data sources were used to inform four types of validity evidence: content, response process, internal structure, and consequential. STEM experts and typical assessment users provided support for content, response process, and conse- quential validity evidence through focus group interviews and cognitive interviews. Psychometric findings suggest strong internal validity evidence for the assessment through high measures of internal consistency reliability, intra‐rater agreement, and test–retest reliability measures. KEYWORDS 21st Century Skills, assessment, validation process