Proceedings of the 2003 Winter Simulation Conference S. Chick, P. J. Sánchez, D. Ferrin, and D. J. Morrice, eds. EXPERIMENTAL DESIGN FOR SIMULATION W. David Kelton Department of Quantitative Analysis and Operations Management University of Cincinnati Cincinnati, OH 45221-0130, U.S.A. Russell R. Barton The Smeal College of Business Administration The Pennsylvania State University University Park, PA 16802, U.S.A. ABSTRACT • How many runs should you make? • How should you interpret and analyze the output? This tutorial introduces some of the ideas, issues, chal- lenges, solutions, and opportunities in deciding how to ex- periment with simulation models to learn about their be- havior. Careful planning, or designing, of simulation experiments is generally a great help, saving time and ef- fort by providing efficient ways to estimate the effects of changes in the model’s inputs on its outputs. Traditional experimental-design methods are discussed in the context of simulation experiments, as are the broader questions pertaining to planning computer-simulation experiments. • What’s the most efficient way to make the runs? These questions, among others, are what you deal with when trying to design simulation experiments. My purpose in this tutorial is to call your attention to these issues and indicate in general terms how you can deal with them. I won’t be going into great depth on a lot of technical details, but refer you instead to any of several texts on simulation that do, and to tutorials and reviews on this subject in this and recent Proceedings of the Winter Simulation Conference. General book-based references for this subject include chapter 12 of Law and Kelton (2000), chapter 11 of Kelton, Sadowski, and Sadowski (2002), Banks, Carson, Nelson, and Nicol (2001), Kleijnen (1998), and Barton (1999), all of which contain numerous refer- ences to other books and papers on this subject. Examples of application of some of these ideas can be found in Hood and Welch (1992, 1993) and Swain and Farrington (1994), and another recent tutorial is Barton (2002). Parts of this paper are taken from Kelton (1997, 2000), which also con- tain further references and discussion on this and closely related subjects. 1 INTRODUCTION The real meat of a simulation project is running your mod- els and trying to understand the results. To do so effec- tively, you need to plan ahead before doing the runs, since just trying different things to see what happens can be a very inefficient way of attempting to learn about your models’ (and hopefully the systems’) behaviors. Careful planning of how you’re going to experiment with your models will generally repay big dividends in terms of how effectively you learn about the systems and how you can exercise your models further. 2 WHAT IS THE PURPOSE OF THE PROJECT? This tutorial looks at such experimental-design issues in the broad context of a simulation project. The term “ex- perimental design” has specific connotations in its traditional interpretation, and I will mention some of these below, in Section 5. But I will also try to cover the issues of planning your simulations in a broader context, which consider the special challenges and opportunities you have when con- ducting a computer-based simulation experiment rather than a physical experiment. This includes questions of the over- all purpose of the project, what the output performance measures should be, how you use the underlying random numbers, measuring how changes in the inputs might affect the outputs, and searching for some kind of optimal system configuration. Specific questions of this type might include: Though it seems like pretty obvious advice, it might bear mentioning that you should be clear about what the ultimate purpose is of doing your simulation project in the first place. Depending on how this question is answered, you can be led to different ways of planning your experiments. Worse, failure to ask (and answer) the question of just what the point of your project is can often leave you adrift without any organized way of carrying out your experiments. For instance, even if there is just one system of interest to analyze and understand, there still could be questions like run length, the number of runs, allocation of random numbers, and interpretation of results, but there are no questions of which model configurations to run. Likewise, if there are just a few model configurations of interest, and they have been given to you (or are obvious), then the • What model configurations should you run? • How long should the runs be? 59