Proceedings of the 2007 Winter Simulation Conference S. G. Henderson, B. Biller, M.-H. Hsieh, J. Shortle, J. D. Tew, and R. R. Barton, eds. RANKING AND SELECTION TECHNIQUES WITH OVERLAPPING VARIANCE ESTIMATORS Christopher Healey David Goldsman Seong-Hee Kim The H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332-0205, U.S.A. ABSTRACT Some ranking and selection (R&S) procedures for steady- state simulation require an estimate of the asymptotic vari- ance parameter of each system to guarantee a certain proba- bility of correct selection. We show that the performance of such R&S procedures depends on the quality of the variance estimates that are used. In this paper, we study the perfor- mance of R&S procedures with two new variance estimators — overlapping area and overlapping Cram´ er-von Mises es- timators — which show better long-run performance than other estimators previously used in R&S problems. 1 INTRODUCTION In ranking and selection (R&S), we are concerned with the selection of the best system out of a number of alternatives. We also require a certain probability of correct selection (PCS) in our procedures. In steady-state simulation, we are usually interested in determining the system that has either the largest or smallest expected value of a specific steady-state performance measure. Many R&S procedures have been developed assum- ing that basic observations are independent and identically distributed (i.i.d.) normal random variates. Those R&S procedures can be used for steady-state simulation, if an experimenter is willing to use as basic observations within- replication averages from multiple replications (after dele- tion of initial data) or batch means from a single replication. However, Goldsman et al. (2002) and Kim and Nelson (2006) found that both approaches could diminish the efficiency of fully sequential procedures and proposed two procedures that take individual observations (such as consecutive wait times) as basic observations from a single replication. It is often the case that individual observations from steady-state simulations possess an inherent dependence structure, and thus the usual marginal variance is not a good measure for the variability of such dependent data. Instead, most selection procedures require estimates for the so-called variance parameters of the competitors, which are unknown in typical simulation applications; the variance parameter for a particular steady-state process is simply the sum of the covariances at all lags. For instance, the procedures due to Goldsman et al. (2002) and Kim and Nelson (2006) — called R+, KN+, and KN++ — use well- known estimators for the variance parameter that happen to be asymptotically chi-squared distributed. A number of new variance parameter estimators have recently been developed in the literature. For example, Alexopoulos et al. (2006b) proposed various overlapping standardized time series (STS) estimators. These overlap- ping STS estimators have smaller asymptotic variance and smaller bias compared to their non-overlapping counterparts. As better variance estimators are introduced, one might be- come interested in whether these new variance estimators can be incorporated into R&S procedures with beneficial results in terms of the required number of observations and the attained probability of correct selection. In the current paper, we investigate such issues. This paper is organized as follows: Section 2 defines notation and introduces the variance estimators considered herein. Section 3 gives an overview of three R&S procedures specifically designed for steady-state simulation. In Sections 4 and 5, we discuss our experimental setup and results, followed by conclusions in Section 6. 2 VARIANCE ESTIMATORS This section describes the notation used throughout the paper and introduces the variance estimators that we will implement in the selection procedures. 2.1 Notation Let Y i ≡{Y ij : j = 1,..., m} be a realization from a single replication of a simulation of system i = 1,..., k. For exam- ple, Y ij could be the jth individual waiting time in the ith queueing system under consideration. After deleting some 522 1-4244-1306-0/07/$25.00 ©2007 IEEE