Supplementary materials for this article are available at http://pubs.amstat.org/toc/tech/51/4. Bayesian Validation of Computer Models Shuchun WANG School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332 (swang1@isye.gatech.edu) Wei CHEN Department of Mechanical Engineering Northwestern University Chicago, IL 60208 (weichen@northwestern.edu) Kwok-Leung TSUI School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332 (ktsui@isye.gatech.edu) Computer models are mathematical representations of real systems developed for understanding and in- vestigating the systems. They are particularly useful when physical experiments are either cost- prohibitive or time-prohibitive. Before a computer model is used, it often must be validated by comparing the com- puter outputs with physical experiments. This article proposes a Bayesian approach to validating com- puter models that overcomes several difficulties of the frequentist approach proposed by Oberkampf and Barone. Kennedy and O’Hagan proposed a similar Bayesian approach. A major difference between their approach and ours is that theirs focuses on directly deriving the posterior of the true output, whereas our approach focuses on first deriving the posteriors of the computer model and model bias (difference be- tween computer and true outputs) separately, then deriving the posterior of the true output. As a result, our approach provides a clear decomposition of the expected prediction error of the true output. This de- composition explains why and how combining computer outputs and physical experiments can provide more accurate prediction compared with using only computer outputs or only physical experiments. Two examples are used to illustrate our proposed approach and compare it with the approach Kennedy and O’Hagan. This article has supplementary material online. KEY WORDS: Bayesian; Computer model; Gaussian process; Model bias; Model validation; Physical experiments. 1. INTRODUCTION Computer models are mathematical representations of real systems, for example, a group of partial differential equations with initial and boundary conditions for engineering problems. They are commonly used to investigate complex systems for which physical experiments are highly expensive or overly time-consuming (Sacks et al. 1989; Welch et al. 1992; Santner, Williams, and Notz 2003). Before using a computer model to investigate a real system, however, one needs to address an im- portant question: “How well does the computer model represent the real system?” Without a meaningful answer to this ques- tion, any conclusions based on the analysis of outputs from a computer model are only about that particular computer model and cannot be simply extended to the real system of interest. The process of determining to what degree a computer model accurately represents the real system, known as model valida- tion (American Institute of Aeronautics and Astronautics 1998), generally involves the comparison of outputs computed from a computer model to observations collected from physical exper- iments. Model validation should not be confused with model verifi- cation. Model verification is defined as “the process of deter- mining that a model implementation accurately represents the developer’s conceptual description of the model and the solu- tion to the model” (American Institute of Aeronautics and As- tronautics 1998). In this article we focus on model validation, assuming that the computer model has been verified. General discussions on model verification and validation have been pro- vided by Roache (1998), Oberkampf and Trucano (2000), Sant- ner, Williams, and Notz (2003), and Oberkampf, Trucano, and Hirsch (2004). Computer outputs and physical observations for validating a computer model can be compared in many different ways. For example, one can graphically display both computer outputs and physical observations in one plot and see how the com- puter outputs agree with the physical observations. The graph- ical comparison is simple and easy to use and probably is the first thing that should be done before any sophisticated methods are attempted. But such an approach is obviously too subjective and does not provide a quantitative measure on how well the computer model represents the real system. An alternative ap- proach for validating computer models is to formulate model validation as a hypothesis testing problem (Hills and Trucano 1999, 2002; Hills 2006). Hills and Trucano (2002) proposed a χ 2 test for validating computer models. They assumed that the vector of computer outputs and that of physical observations both follow independent multivariate normal distributions, and computed a χ 2 statistic to test a null hypothesis that the mean of the difference of the two vectors (the model bias) is 0. © 2009 American Statistical Association and the American Society for Quality TECHNOMETRICS, NOVEMBER 2009, VOL. 51, NO. 4 DOI 10.1198/TECH.2009.07011 439 Downloaded by [107.10.148.4] at 12:07 10 December 2012