H. Sarin M. Kokkolaras G. Hulbert P. Papalambros Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109-1316 S. Barbat R.-J. Yang Passive Safety, Research and Advanced Engineering, Ford Motor Company, Highland Park, MI 48203-3177 Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics Computer modeling and simulation are the cornerstones of product design and develop- ment in the automotive industry. Computer-aided engineering tools have improved to the extent that virtual testing may lead to significant reduction in prototype building and testing of vehicle designs. In order to make this a reality, we need to assess our confi- dence in the predictive capabilities of simulation models. As a first step in this direction, this paper deals with developing measures and a metric to compare time histories ob- tained from simulation model outputs and experimental tests. The focus of the work is on vehicle safety applications. We restrict attention to quantifying discrepancy between time histories as the latter constitute the predominant form of responses of interest in vehicle safety considerations. First, we evaluate popular measures used to quantify discrepancy between time histories in fields such as statistics, computational mechanics, signal pro- cessing, and data mining. Three independent error measures are proposed for vehicle safety applications, associated with three physically meaningful characteristics (phase, magnitude, and slope), which utilize norms, cross-correlation measures, and algorithms such as dynamic time warping to quantify discrepancies. A combined use of these three measures can serve as a metric that encapsulates the important aspects of time history comparison. It is also shown how these measures can be used in conjunction with ratings from subject matter experts to build regression-based validation metrics. DOI: 10.1115/1.4002478 1 Introduction Automotive manufacturers have to meet several vehicle safety regulations and mandatory Federal Motor Vehicle Safety Stan- dards FMVSS. Additionally, consumer information programs such as the new car assessment program NCAP and the Insur- ance Institute of Highway Safety IIHS impose further require- ments on vehicle safety. Currently, assessment of whether these requirements are satisfied is conducted through numerous, costly and time-consuming physical experiments. Computer modeling and simulation-based methods for virtual vehicle safety analysis and design verification could make this process more time and cost efficient. Moreover, virtual testing VT can improve real-world vehicle safety beyond regulatory requirements since computer predictions can be used to extend the range of protection to real-world crash conditions at speeds and configurations not addressed by current regulations. To achieve the promises of VT, computer predictions need veri- fication and validation V&V, so that the designs obtained using simulation models can be cleared for production with minimal or reduced physical prototype testing. The American Institute of Aeronautics and Astronautics guide for verification and validation of computational fluid dynamics simulations defines verification and validation as follows 1. “Verification is the process of determining that a model implementation accurately represents the developer’s con- ceptual description of the model and the solution to the model.” “Validation is the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model.” The American Society of Mechanical Engineers Standards Committee on verification and validation in computational solid mechanics describes model validation as a two-step process 2: 1. quantitatively comparing the computational and experimen- tal results for the response of interest 2. determining whether there is an acceptable agreement be- tween the model and the experiment for the intended use of the model Oberkampf and Barone proposed in Ref. 3 six properties that a validation metric should satisfy. These six properties form a generic guideline and act as a set of requirements for the devel- opment of a new validation metric. Their third property dictates that an effective metric for measuring the discrepancy between simulation model responses represented by time histories is nec- essary to accomplish the first step of the validation process. In this paper, we review existing error measures and metrics and discuss their advantages and limitations. We then propose a combination of measures associated with three physically meaningful error characteristics: phase, magnitude, and slope. The proposed ap- proach utilizes measures such as cross-correlation and L 1 norm and algorithms such as dynamic time warping DTW to quantify the discrepancy between time histories. We then show how these measures can be used to build regression-based validation metrics in cases where subject matter expert data are available. It is important to note that four of the remaining five properties advocated by Oberkampf and Barone 3 for useful validation metrics involve the uncertainties related to numerical error, ex- perimental error, experiment postprocessing, and the number of experiments conducted. While these are critical issues to be ad- dressed, they are not considered in this paper as the goal of this present work is to establish an appropriate set of error measures for vehicle safety applications and to assess combinations of these measures into an error metric. With an established set of error measures, the next step toward a fully developed validation metric is to use the error measures to provide the quantitative values for Contributed by the Dynamic Systems Division of ASME for publication in the JOURNAL OF DYNAMIC SYSTEMS,MEASUREMENT, AND CONTROL. Manuscript received September 15, 2008; final manuscript received May 12, 2010; published online October 28, 2010. Assoc. Editor: Jeffrey L. Stein. Journal of Dynamic Systems, Measurement, and Control NOVEMBER 2010, Vol. 132 / 061401-1 Copyright © 2010 by ASME Downloaded 28 Oct 2010 to 141.212.126.69. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm