4 Evaluation Journal of Australasia Vol 17 | No 3 | 2017 ACADEMIC ARTICLE Evaluation Journal of Australasia Vol 17 | No 3 | 2017 | pp. 4–13 RALPH RENGER | JIRINA FOLTYSOVA | JESSICA RENGER | WAYNE BOOZE Defning systems to evaluate system efciency and efectiveness This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The frst foundational step is to defne the system. Systems thinking provides theoretical rationale for defning the system boundaries, components, and relationships. However, there is no literature describing how to defne these system elements. Using an example from the evaluation of several United States cardiac care systems, the paper shares a number of methods used to defne the system boundaries, components, and relationships. The paper describes how each of these elements informs the evaluation of step two of SET—evaluating system efciency. The discussion shares lessons learned, and notes the relationship between methods used in system and program evaluation. Defning systems to evaluate system efciency and efectiveness The interest in the potential of systems thinking to improve evaluation quality continues to grow (Adams, Hester, Bradley, Meyers & Keating, 2014; Renger, 2015; Renger, 2016; Renger, Foltysova, Ienuso, Renger & Booze, 2017; Renger, McPherson, Kontz-Bartels & Becker, 2016; Renger, Wood, Williamson & Krapp, 2012; Rogers, 2011; Wehipeihana, 2011; Williams, 2015; Williams & Hummelbrunner, 2010). One reason for the interest in systems thinking is its potential to address limitations associated with theory-driven program evaluation (Renger, 2015; Renger, 2016). Critics of theory-driven program evaluation argue approaches, like logic modeling, are artifcial because program assumptions tend to be linear. As such, the program assumptions do not refect the reality in which programs operate, ignoring many other contextual factors infuencing program outcomes (Morrel, 2010). Thus, fndings from such evaluations are sometimes difcult to interpret, often meaningless, and therefore, are of limited utility (Lee, 2017; Patton, 2008). Systems thinking represents a way forward to address these criticisms and produce more usable evaluations (Patton, 2008; Williams, 2015). Within the evaluation literature, three core systems principles continually emerge: boundaries, components, and relationships (Hargreaves & Podems, 2012; Williams & Hummelbrunner, 2010). It is reasoned the application of these core principles in program evaluation should lead to better understanding of the context in which a program operates and thus more realistic and useful evaluations (Patton, 2008). Williams and Hummelbrunner (2010) published numerous research methods to assist evaluators in applying systems thinking principles to improve program evaluations. More recently, Renger (2015) examined the utility of systems thinking to evaluate modern day systems, not programs. Ericson (2011) defnes a modern day system as an integrated composite of components that provide function and capability to satisfy a stated need or objective. A system is a holistic unit that is greater than the sum of its parts. It has structure, function, behavior, characteristics, and interconnectivity. Modern day systems are typically composed of people, products, and environments that together generate complexity and capability (p. 402).