Thermal, Power, and Co-location Aware Resource Allocation in Heterogeneous High Performance Computing Systems Mark A. Oxley * , Eric Jonardi * , Sudeep Pasricha *† , Anthony A. Maciejewski * , Gregory A. Koenig ‡ , and Howard Jay Siegel *† * Department of Electrical and Computer Engineering † Department of Computer Science Colorado State University, Fort Collins, Colorado 80523, USA ‡ Oak Ridge National Laboratory Oak Ridge, Tennessee 37830, USA {mark.oxley,eric.jonardi,sudeep,aam,hj}@colostate.edu, koenig@ornl.gov Abstract—The rapid increase in power consumption of high performance computing (HPC) systems has led to an increase in the amount of cooling resources required to operate these facilities at a reliable threshold. The cooling systems contribute a large portion of the total power consumption of the facility, thus driving up the costs of providing power to these facilities. In addition, when cores sharing resources (e.g., last-level cache) execute applications at the same time, they can experience contention and therefore performance degradation. By taking a holistic approach to HPC facility management through intelligently allocating both computing and cooling resources, the performance of the HPC system can be maximized by considering co-location while obeying power consumption and thermal constraints. The performance of the system is quantified as the total reward earned from completing tasks by their individual deadlines. We propose three novel resource allocation techniques to maximize performance under power and thermal constraints when considering co-location effects: (1) a greedy heuristic, (2) a genetic algorithm technique used in combination with a new local search technique that guarantees the power and thermal constraints, and (3) a non- linear programming based approach (from previous work), adapted to consider co-location effects. Keywords-heterogeneous computing; resource management; thermal-aware computing; power-aware computing; data cen- ter; DVFS; memory interference I. I NTRODUCTION The power consumption of high-performance comput- ing (HPC) systems and data centers is increasing rapidly, leading to an increase in the amount of cooling resources required to operate these HPC systems at a safe temperature threshold. The increase of power consumption to operate today’s facilities has led to the design of power-efficient systems such as the TSUBAME-KFC at the Tokyo Institute of Technology that tops the Green500 list with an efficiency of 4.5 GFLOPS/Watt [1]. Extrapolating the power consump- tion of the TSUBAME-KFC system to exascale, it would consume 222 MW, which equates to approximately $145 million per year in electricity costs in the United States. The Defense Advanced Research Projects Agency (DARPA) has set the target for an exaflop system to consume no more than 20 MW [2], an order of magnitude of power consumption lower than what can be achieved using today’s computing infrastructure. One technique that can be used to reduce the power consumption and also manage thermal issues in large-scale computing systems, such as data centers, is power-aware and thermal-aware resource allocation. This approach involves exploiting system characteristics such as the heterogeneity among different servers (offering different levels of per- formance and power consumption), dynamic voltage and frequency scaling (DVFS) in cores, and the interactions between temperatures of the compute nodes and computer room air conditioning (CRAC) units. By distributing work- loads, configuring DVFS, and setting CRAC thermostats in an intelligent manner, it becomes possible to efficiently bal- ance the compute performance with the power consumption and temperature profiles of the HPC facility. By the law of conservation of energy, the power consumed by servers is dissipated as heat. This heat must be removed by the cooling infrastructure so that compute nodes can safely operate beneath their specified redline temperatures (the maximum safe operating temperatures). The more power that is consumed by compute nodes, the more power CRAC units must consume to remove additional heat and maintain the redline temperatures. Cores within compute nodes are DVFS-enabled, with performance states (P-states) that provide a trade-off between compute performance and power consumed by each core. By intelligently configuring P-states of cores, power consumption can be reduced at the compute nodes, which in turn results in less power that is needed by the CRAC units to cool the facility. As the number of cores increase in emerging multicore processors, access contention in shared memory (e.g., last- level caches, DRAM) can have a pronounced impact on the 978-1-4799-6177-1/14/$31.00 c 2014 IEEE