PROVUS’ DISCREPANCY EVALUATION OF THE DRIVESMART NOVICE DRIVER CD-ROM TRAINING PRODUCT Michael A. Regan, Thomas J. Triggs, Eve Mitsopoulos, Chantel C. Duncan, Stuart T. Godley Monash University Accident Research Centre Phil Wallace Learning Systems Analysis Pty Ltd ABSTRACT The Monash University Accident Research Centre (MUARC) recently completed a research program culminating in the development of a CD-ROM based training product, known as DriveSmart, designed to accelerate in young novice drivers the development of perceptual and cognitive skills known to be important in reducing crash risk. While the use of traditional research strategies to evaluate a training product can provide important measures of instructional effectiveness, these strategies do not necessarily provide the type of information needed to identify and remedy specific flaws in the product. Provus’ Discrepancy Approach is an alternative strategy for evaluating educational programs where performance standards are established, evidence of compliance with these standards is gathered, discrepancies with standards are identified, and corrective actions are taken. This paper discusses the application of Provus’ Discrepancy Approach to the evaluation of DriveSmart, focussing on the identification of discrepancies with the standards, and the changes which were made to the product as a result. INTRODUCTION In 1995, MUARC was contracted by the Victorian Transport Accident Commission (TAC) to conduct research using an advanced driving simulator, to investigate and to determine techniques for effectively training four skills which were identified (Triggs, 1994) as critical in moderating the crash involvement of novice drivers. The four skills identified were: risk perception (the ability to detect, perceive and assess the degree of risk associated with actual and emerging traffic hazards); attentional control (the ability to prioritise attention); time-sharing (the ability to share limited attention between multiple competing driving tasks); and calibration (the ability to moderate task demands according to one’s own performance capabilities). In 1999, this research (see Triggs and Regan,1998) culminated in the development of a CD-ROM training product known as DriveSmart (see Regan, Triggs and Wallace, 1999 for a full description of the product). The content for DriveSmart was drawn from the road safety literature and from the findings of the research program referred to above. Content areas drawn from the literature included: insight training: optimism, commentary driving; prediction; and situation awareness. The remaining content areas were derived from the simulator experiments conducted by MUARC. The approach of Incremental Transfer Learning (ITL; Wallace and Regan, 1998) was selected as the general instructional strategy underpinning the product. ITL places considerable importance on the need to plan for both near-transfer and far- transfer of skills. DriveSmart was launched in Victoria in May 2000. Since its release to the public, in July 2000, over 10,000 copies have been distributed free of charge to learner drivers and various learning and other institutions in Victoria. The purpose of this paper is to report on an evaluation of the CD ROM which was conducted by MUARC under the direction of an independently appointed Project Advisory Committee prior to the product’s release. The evaluation had two major components. The first of these is described in a companion paper presented at this conference (Regan, Triggs & Godley, 2000) that involved an experiment, using an advanced driving simulator, to evaluate the instructional effectiveness of the CD ROM product. The second component, using Provus’ Discrepancy model for curriculum evaluation (Brady, 1983), is the subject of the present paper. Methods which are traditionally used to evaluate products, such as the simulator-based evaluation, do not provide a systematic and comprehensive means for identifying flaws that may be contained within the product. The Discrepancy Model for Curriculum Evaluation, developed by Provus (see Brady, 1983), however, provides a mechanism for doing so. Using this model, specific flaws in a given learning program can be identified and rectified by determining any discrepancies that exist between a pre-determined set of standards for the learning program and what actually occurs in the program. According to this model there are essentially six steps that need to be completed in order to conduct such an evaluation: 1. develop a list of standards which specify the characteristics of ideal implementation of a learning program; 2. determine the information required to compare actual implementation with the defined standards; 3. design methods to obtain the required information;