Session 13c1 0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29 th ASEE/IEEE Frontiers in Education Conference 13c1-7 Establishing the Trustworthiness of Scenario Assignments as Assessment Tools for Undergraduate Engineering Education Flora McMartin, Ann McKenna and Ken Youssefi University of California at Berkeley, Berkeley, CA 94720 Abstract - How to establish the trustworthiness of qualita- tive tools to assess students' knowledge of engineering practices, teamwork, and problem solving is a question that remains unanswered for many faculty. This paper describes a process that was conducted to ensure the dependability of scenario assignments. Scenarios are a qualitative perform- ance assessment tool designed to assess students' knowl- edge of engineering practices, teamwork, and problem solv- ing. Two scoring rubrics, a holistic and an analytic, were developed to assess students' knowledge with respect to the learning outcomes associated with the scenario tool. Care- ful training of the scorers, analysis of the scores, and com- parison of test scores determined dependability of the tool. Students were given a scenario that describes a "day in the life" problem faced by engineers, then they are asked to explain the process they would use to solve the problem. Initial findings suggest that faculty who score the scenarios with the analytic rubric can quickly identify students' strengths and weaknesses in these areas and adapt their course to address the areas where students need attention. At the departmental or college level, scoring the scenarios with the holistic rubric is useful to assess changes in stu- dents' learning and development over time. Careful testing of the rubric revealed that the scores can reliably differen- tiate among student abilities. I. Introduction An unanswered question for many faculty when considering the use of a qualitative assessment tool is how to know when an assessment process that relies on written student essay responses is trustworthy -- that is, that it can de- pendably measure a student learning outcome or objective. Faculty and administrators depend heavily upon question- naires and surveys of student opinion or self-assessment as measures of student learning [1, 2]. Additional measures include tests, which are primarily multiple choice. These measures, however, do not demonstrate a student’s actual ability to accomplish a particular outcome; instead they re- flect a (frequently) inaccurate perception of ability, or knowledge about a particular concept [3]. Unlike these measures, performance assessments are designed to get at the more difficult aspects of higher learning by solving real- istic or authentic problems [4, 5]. Student performance can best be measured by observation; however, this method is time consuming and costly. The alternative is to examine student written work, e.g., design assignments and/or port- folios as evidence of a student’s ability to accomplish spe- cific learning outcomes. When developing a new tool, such as scenarios, it is imperative that the developers establish it’s trustworthiness. In the case of a tool designed to elicit a written response from students, the scoring mechanism and process must also be rigorously tested in order to support the tool’s trust- worthiness [6, 7]. Various methods for testing for trustwor- thiness include quantitative statistical techniques such as factor analysis, i.e., the correlation among questions regard- ing the same dimension, or test – retest, i.e., consistency of scores over time [8]. Performance assessments such as the scenario assignments require somewhat different methods because the student responses reflect creative thinking and originalityno two responses will be the same. So, in the case of the scenario assignments it becomes necessary to combine the more qualitative assessment methods employed in writing and composition with quantitative methods. In addition, the evaluation of the scenario assignment requires careful training of the scorers, analysis of the scores, and comparison of test scores across different versions of the tool [9, 10, 11]. The purpose of this paper is to explain one process used by the assessment team of the Synthesis Coalition to test the credibility and dependability of student written responses to a scenario assignment. Scenario assignments are a qualita- tive assessment tool designed to assess students' knowledge of engineering practices, teamwork, and problem solving. Rather than solving the technical or design problem pre- sented in the scenario, students are asked to describe how they would go about finding the solution to the technical problem with a team of engineers. II. Background The Tool Students respond to an open-ended problem description that outlines a situation and the context for the problem. Stu- dents take on the role of a project manager who must pre- pare a team to tackle the project while he or she is out of town on business. Scenarios are usually completed as homework assignments. Typically, students take between one and two hours writing their response.