Proceedings of the 2005 Winter Simulation Conference M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines, eds. STATISTICAL SELECTION OF THE BEST SYSTEM David Goldsman Seong-Hee Kim School of Industrial & Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, U.S.A. Barry L. Nelson Department of Industrial Engineering and Management Sciences Northwestern University Evanston, IL 60208-3119, U.S.A. ABSTRACT This tutorial discusses some statistical procedures for select- ing the best of a number of competing systems. The term “best” may refer to that simulated system having, say, the largest expected value or the greatest likelihood of yielding a large observation. We describe various procedures for finding the best, some of which assume that the underly- ing observations arise from competing normal distributions, and some of which are essentially nonparametric in nature. In each case, we comment on how to apply the above procedures for use in simulations. 1 INTRODUCTION Experiments are often performed to compare two or more system designs in order to determine which scenario is the best. The statistical methods of screening, selection, and multiple comparisons are applicable when we are interested in making comparisons among a finite, possible large, num- ber of scenarios. The particular method that is appropriate depends on the type of comparison desired and properties of the data under study. For instance, are we interested in comparing means or quantiles? Are the available data independent or correlated within and/or among systems? In this review, the term “best” may refer to that simulated system having, say, the largest expected value or the greatest likelihood of yielding a large observation. We will typically, but not always, regard the best population as the one having the largest expected value. We describe a number of procedures for finding the best, some of which assume that the underlying observations arise from competing normal distributions, and some of which are essentially nonparametric in nature. In each case, we comment on how to apply the above procedures for use in simulations. To get things going, the next section will give some ad- ditional low-level background on screening, selection, and multiple comparisons procedures. Section 3 establishes rel- evant notation and ground rules, while Section 4 presents some very basic methods for purposes of motivating the upcoming procedures. Section 5 discusses three normal means procedures for selecting the best (or nearly the best) scenario. Of the normal procedures, the first is a screen-and- select procedure for finding the population with the largest expected value; in this procedure, inferior competitors are screened out after an initial stage of sampling. The second is a sequential procedure that can eliminate inferior choices at any stage and uses the common random numbers variance reduction technique in which we intentionally induce posi- tive correlation between scenarios. The third is an efficient two-stage procedure that also uses common random num- bers. Section 6 deals with three nonparametric procedures. Of these procedures, the first is a single-stage procedure for finding the most probable multinomial cell, the second is sequential, and the third is a clever augmentation that makes more efficient use of the underlying observations. We give conclusions in Section 7. There are a number of general references for the inter- ested reader in this area of selection of the best. Gibbons, Olkin, and Sobel (1977) and Bechhofer, Santner, and Golds- man (1995) give presentations from a statistical point of view, while Goldsman and Nelson (1998) and Kim and Nel- son (2005a) devote a great deal of effort to the simulation side of the story. 2 BACKGROUND We will usually assume that the observations coming from a particular scenario are independent and identically dis- tributed (i.i.d.). Since this is never the case when dealing with simulation output (which is, for instance, almost always serially correlated), we will make appropriate comments to show how to apply the above procedures for use in simu- lations. What are screening, selection, and multiple comparisons procedures? Screening and selection procedures (SSPs) are statistical methods designed to find the “best” (or “nearly the best”) system from among a collection of competing alter- 178