Self-Adaptive Scheduler Parameterization via Online Simulation * Barry Lawson Department of Math and Computer Science University of Richmond Richmond, VA 23173, USA blawson@richmond.edu Evgenia Smirni Department of Computer Science College of William and Mary Williamsburg, VA 23187-8795, USA esmirni@cs.wm.edu Abstract Although thoroughly investigated, job scheduling for high-end parallel systems remains an inexact science, re- quiring significant experience and intuition from system administrators to properly configure batch schedulers. Pro- duction schedulers provide many parameters for their con- figuration, but tuning these parameters appropriately can be very difficult — their effects and interactions are of- ten nonintuitive. In this paper, we introduce a methodology for automating the difficult process of job scheduler pa- rameterization. Our proposed methodology is based on us- ing past workload behavior to predict future workload, and on online simulations of a model of the actual sys- tem to provide on-the-fly suggestions to the scheduler for automated parameter adjustment. Detailed perfor- mance comparisons via simulation using actual supercom- puting traces indicate that out methodology consistently outperforms other workload-aware methods for sched- uler parameterization. Keywords: batch scheduler parameterization, high-end parallel systems, self-adaptive schedulers, backfilling, per- formance analysis, online simulation. 1. Introduction Large-scale clusters of multiprocessors have emerged as the dominant platform for high-end computing, comprising almost 60% of the Top 500 list (www.top500.org) as of November 2004.Traditional scheduling policies for clus- ters and other high-end distributed memory systems con- sider scheduling a single resource, i.e., CPU only, and fo- cus on treating differently interactive versus batch jobs [1] with the goal of maximizing system utilization. Industrial- strength schedulers that are widely accepted by the super- * This work was partially supported by the National Science Foundation under grants CCR-0098278, ACI-0090221, and ITR-0428330. computing community, including the Maui scheduler [11], PBS [13], and IBM LoadLeveler [4], offer a variety of con- figuration parameters that allow the system administrator to customize the scheduling policy according to the site’s needs. The effectiveness of the scheduler is directly deter- mined by the administrator’s choices for the configuration parameters. However, tuning these parameters to optimize performance is extremely challenging because it is difficult to predict the outcome based on the interaction of many dif- ferent changing parameters. Choosing appropriate schedul- ing parameters in such complex environments requires not only awareness of the performance effects of existing queu- ing in the system, but also requires a keen understanding of the effects of transient behavior in the workload. Static definition of scheduling parameters clearly is insufficient to cope with sudden changes in the workload intensities and demands. In the literature, scheduling policies based on backfill- ing have been proposed as more efficient alternatives to simple FCFS scheduling [12]. In backfilling, users are ex- pected to provide nearly accurate estimates of the job ex- ecution times. Using these estimates, the scheduler scans and reorders the waiting queue, allowing certain short jobs to surpass long jobs in the queue provided those short jobs do not delay certain previously submitted jobs. The goal of backfilling is to decrease system fragmentation and increase system utilization [12, 15] by using otherwise idle proces- sors for immediate execution. Various versions of backfill- ing have been proposed [6, 12, 14], including algorithms for gang-scheduling [16]. Backfilling can be found in most con- temporary cluster installation schedulers [2, 5]. For a recent survey of parallel job scheduling in general, including back- filling in particular, we direct the interested reader to [2]. The two important external parameters that affect perfor- mance and scheduling decisions in a parallel system are the arrival process and the service process. Variations in the in- tensities of arrivals and service requirements are responsible for the growth of waiting queues to a certain point beyond the “knee of the curve”, i.e., the point where the individual 0-7695-2312-9/05/$20.00 (c) 2005 IEEE