Proceedings of the 2009 Winter Simulation Conference M. D. Rossetti, R. R. Hill, B. Johansson, A. Dunkin, and R. G. Ingalls, eds. DO MEAN-BASED RANKING AND SELECTION PROCEDURES CONSIDER SYSTEMS’ RISK? Demet Batur F. Fred Choobineh Department of Industrial and Management Systems Engineering University of Nebraska-Lincoln Lincoln, NE 68588, U.S.A. ABSTRACT The legacy simulation approach in ranking and selection procedures compares systems based on a mean performance metric. The best system is most often deemed as the one with the largest (or smallest) mean performance metric. In this paper, we discuss the limitations of the mean-based selection approach. We explore other selection criterion and discuss new approaches based on stochastic dominance using an appropriate section of the distribution function of the performance metric. In this approach, the decision maker has the flexibility to determine a section of the distribution function based on the specific features of the selection problem representing either the downside risk, upside risk, or central tendency of the performance metric. We discuss two different ranking and selection procedures based on this new approach followed by a small experiment and present some open research problems. 1 INTRODUCTION Discrete-event simulation is used in the analysis and comparison of complex stochastic systems, e.g., manufacturing systems and communications networks. Stochastic systems are compared based on one or more performance metrics of interest that are outputs of simulations models. Since simulation outputs are random, statistically-sound ranking and selection (R&S) techniques are needed to choose the best system. R&S procedures are a collection of statistical procedures for comparing a finite set of stochastic systems with the goal of finding the best among them. The legacy approach in R&S procedures compares systems based on the mean dominance of a performance metric of interest. Recent developments in the mean-based R&S procedures when the samples are independent and identically (IID), normally distributed are the works of Kim and Nelson (2001), Hong (2006), Pichitlamken et al. (2006) , Chick and Inoue (2001), and Chen et al. (2000). Kim and Nelson (2001) propose a fully-sequential procedure KN that obtains observations from each system one at a time until there is enough evidence that a system’s mean is dominated by one of the others. The ultimate objective is to select a system with a guarantee on the probability of correct selection. Hong (2006) presents a computationally more efficient version of the KN procedure where new samples are allocated to systems according to their variances. Pichitlamken et al. (2006) propose a fully-sequential mean-based procedure for evaluating neighborhood solutions in a simulation optimization algorithm when partial or complete information on solutions previously visited is maintained. Chen et al. (2000) and Chick and Inoue (2001) use the Bayesian approach to the mean-based selection problem. Their procedures choose the best system (largest or smallest mean) such that the posterior probability of correct selection is maximized while satisfying a simulation budget constraint. An extensive comparison of the mean-based R&S procedures is given in Branke et al. (2007). For a comprehensive review of the R&S procedures in simulation, refer to Kim and Nelson (2006). There are two types of discrete-event simulations: terminating and steady-state. In terminating simulations, the simulation starts at time zero under some well-specified initial conditions, and there is a natural ending event that often specifies the finite horizon that the system operates. Since the interest is on the behavior of the system over a finite-time horizon, the initial conditions would have a large impact on the performance of the system. However, for the steady-state simulations, often the interest is in the most representative behavior of the system over a long period. Since the interest is on the performance of the system over a long-time horizon, the impact of the initial conditions on the performance of the system is negligible. 423 978-1-4244-5771-7/09/$26.00 ©2009 IEEE