This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON EDUCATION 1 Model-Based Systems Thinking: Assessing Engineering Student Teams Rea Lavi , Member, IEEE, Yehudit Judy Dori , Niva Wengrowicz , and Dov Dori , Fellow, IEEE Abstract—Contribution: A rubric for assessing the systems thinking expressed in conceptual models of technological systems has been constructed and assessed using a formal methodology. The rubric, a synthesis of prior findings in science and engineer- ing education, forms a framework for improving communication between science and engineering educators. Background: Systems thinking is an important skill in engi- neering, but to date no rubric for assessing this skill based on a formal methodology has been published with reliability and validity measures. Research Questions: (a) What attributes should a rubric for assessing the systems thinking of engineering students comprise? (b) To what extent can such a rubric serve in assessing the systems thinking level of engineering students, as expressed in their conceptual system models? Methodology: Based on a literature review of systems think- ing assessment in science and engineering education, the authors classified the systems thinking attributes they had identified into system function, structure, and behavior. Scoring instruc- tions were developed for conceptual system models based on Object-Process Methodology ISO 19450, a formal methodology and language for model-based systems engineering. A total of 142 undergraduate engineering students in 32 teams modeled Web-based systems using the methodology. Each team submit- ted two models of the same system, at the middle and end of semester. Models were scored using the rubric, and its reliability and validity were evaluated. Findings: Indications of interrater reliability, internal con- sistency, and construct validity were acceptable, excluding the system function aspect, implying the rubric may be used reliably for its intended purpose. Manuscript received August 2, 2018; revised December 8, 2018, March 5, 2019, June 14, 2019, and August 15, 2019; accepted September 21, 2019. This work was supported by Bernard M. Gordon Center for System Engineering under Grant 2026787. (Rea Lavi, Yehudit Judy Dori, Niva Wengrowicz, and Dov Dori contributed equally to this work.) (Corresponding author: Rea Lavi.) R. Lavi is with the Faculty of Education in Science and Technology, Technion—Israel Institute of Technology, Haifa 3200003, Israel (e-mail: realavi@gmail.com). Y. J. Dori is with the Faculty of Education in Science and Engineering, Technion—Israel Institute of Technology, Haifa 3200003, Israel, and also with the Samuel Neaman Institute, Haifa 3200003, Israel (e-mail: yjdori@technion.ac.il). N. Wengrowicz is with the Faculty of Industrial Engineering and Management, Technion—Israel Institute of Technology, Haifa 3200003, Israel (e-mail: nivawen@technion.ac.il). D. Dori is with the Faculty of Industrial Engineering and Management, Technion—Israel Institute of Technology, Haifa 3200003, Israel, and also with the Systems Design and Management Program, Massachusetts Institute of Technology, Cambridge, MA 02139 USA (e-mail: dori@technion.ac.il). Digital Object Identifier 10.1109/TE.2019.2948807 Index Terms—Assessment tools, conceptual modeling, object- process methodology ISO 19450, rubric, student assessment, systems thinking, student teams, undergraduate. I. I NTRODUCTION A S SYSTEMS are becoming more complex, the require- ment for systems thinking is increasing [1], [2]. Both the Next Generation Science Standards [3] and ABET’s 2018–2019 criteria for accrediting engineering programs [4] include concepts related to systems thinking as part of engineering education. Traditionally, each discipline within science education [5]–[7] and engineering education [8], [9] has described and assessed systems thinking distinctly. With each discipline and sub-discipline having its own insights into systems and systems thinking, the conceptualization of systems thinking in engineering education may benefit from the science education perspective on systems thinking. Moreover, since science is the primary basis for engineer- ing, a systems perspective on science can help develop better understanding of engineered systems. This work aims to facilitate communication between these two related domains. A. Research Objective and Research Questions The objective of this study was to develop a rubric for assessing the systems thinking of engineering students, as expressed in conceptual models they constructed using a for- mal methodology. The resulting research questions were as follows: (a) What attributes should a rubric for assessing the systems thinking of engineering students comprise? (b) To what extent can such a rubric serve in assessing the systems thinking level of engineering students, as expressed in their conceptual system models? B. System Defining the term system has been the subject of numer- ous studies [10]. For the purpose of this study, the following working definition, relevant to both science and engineer- ing education, is used: A system is an entity composed of interacting parts that delivers a function via its architecture, i.e., a combination of its structure and behavior. This function is achieved through interactions of the system parts internally and with the system’s environment. These interactions can be explained by cause and effect relationships. Some system properties vary from those of its individual parts. Finally, arti- ficial (engineered) systems have purpose – a predetermined 0018-9359 c  2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.