Proceedings of the 2000 Winter Simulation Conference J. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick, eds. OUTPUT ANALYSIS PROCEDURES FOR COMPUTER SIMULATIONS David Goldsman School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, U.S.A. Gamze Tokol Earley Corporation 130 Krog St. Atlanta, GA 30307, U.S.A. ABSTRACT This paper concerns the statistical analysis of output from discrete-event computer simulations. In particular, we dis- cuss problems involving terminating simulations, the ini- tialization of simulations, steady-state point and confidence interval estimation for various system parameters, and com- parison among competing system designs. 1 INTRODUCTION Since the input processes driving a simulation are usually random variables (e.g., interarrival times, service times, and breakdown times), a prudent experimenter must also regard the output from the simulation as random. Thus, runs of the simulation only yield estimates of measures of system performance (e.g., the mean customer waiting time). Of course, these estimators are themselves random variables, and are therefore subject to sampling error. This sampling error must be taken into account in a rigorous way if we are to make valid inferences or decisions concerning the performance of the underlying system. The fundamental problem is that simulations almost never produce raw output that is independent and identically distributed (i.i.d.) normal data. For example, consecutive customer waiting times from a complicated queueing system • Are not independent — typically, they are serially correlated. If one customer at the post office waits in line a long time, then the next customer is also likely to wait a long time. • Are not identically distributed. Customers showing up early in the morning might have a much shorter wait than those who show up just before closing time. • Are not normally distributed — they are usually skewed to the right (and are certainly never less than zero). These facts of life make it difficult to apply “classical” statistical techniques to the analysis of simulation output. And so our purpose in this survey is to give practical methods to perform statistical analysis of output from discrete-event computer simulations. In order to facilitate the presentation, we identify two types of simulations with respect to output analysis. 1. Terminating (or transient) simulations. Here, the nature of the problem explicitly defines the length of the simulation run. For instance, we might be interested in simulating a bank that closes at a specific time each day. 2. Nonterminating (steady-state) simulations. Here, the long-run behavior of the system is studied. Presumedly this “steady-state” behavior is inde- pendent of the simulation’s initial conditions. An example is that of a continuously running produc- tion line for which the experimenter is interested in some long-run performance measure. Techniques to analyze output from terminating simulations are primarily based on the method of independent repli- cations, discussed in §2. Additional problems arise for steady-state simulations. One must now worry about the problem of starting the simulation — how should it be initialized at time zero, and how long must it be run before data representative of steady state can be collected? Such initialization problems are considered in §3. Then §4 deals with methods of point and confidence interval estimation for steady-state simulation performance parameters. §5 con- cerns the problem of comparing a number of competing systems, i.e., which is the “best” system? §6 presents con- clusions and provides the interested reader with additional references. Finally, we note that parts of this paper fol- low the discussions in Goldsman (1992) and Goldsman and Tokol (1997). 39