An Energy Case for Hybrid Datacenters Byung-Gon Chun † , Gianluca Iannaccone † , Giuseppe Iannaccone  , Randy Katz ‡ , Gunho Lee ‡ , Luca Niccolini  † Intel Labs Berkeley, ‡ University of California at Berkeley,  University of Pisa ABSTRACT Reducing energy consumption in datacenters is key to building low cost datacenters. To address this challenge, we explore the poten- tial of hybrid datacenter designs that can handle diverse workloads with different service level agreements power efficiently by mixing low power platforms with high performance ones. We evaluate the feasibility of our approach through experiments and then discuss the design challenges and options of hybrid datacenters. 1. INTRODUCTION Energy consumption of the computing infrastructure has become a major concern for industry and society. Today’s datacenters, the backbone of the computing infrastructure, are limited in scale by the costs associated with power (distribution, cooling, density). Studies estimate that power-related costs represent already almost 50% of the operating cost of a datacenter and they are growing faster than compute-related costs (i.e., server and network equip- ment). Energy efficiency is now a first-class design concern at all levels – computation and data processing, power distribution at the rack and server level, power generation and transmission, etc. Companies such as Microsoft and Google are deploying new datacenters near cheap power sources to mitigate energy costs. Pro- cessor manufacturers are pursuing their roadmap of multi-core ar- chitectures [9] and low-power designs [14]. Several research pro- posals deal with power efficient designs and protocols for specific workloads [12], office environments [6, 20] and high speed net- works [19]. In this paper, we look at one specific aspect: energy efficient clusters for large datacenters. As a first step, we consider the cur- rent trends in server designs and try to exploit them to our advan- tage. Traditionally, power efficient designs attempt to find the right balance between two distinct, and often conflicting, requirements: (i) deliver high performance at peak power (i.e., maximize com- pute capacity for a given power budget) and (ii) scale power con- sumption with load (i.e., energy proportionality and very low power operations). A fundamental challenge in finding a good balance is that, when it comes to processor design, the mechanisms that sat- isfy the two requirements above are significantly different. High Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. HotPower’09, October 10, 2009, Big Sky, MT, USA. Copyright 2009 ACM ...$5.00. performance requires mechanisms to mask memory and I/O laten- cies using large multi-level caches (today’s server processors use three cache levels with the last-level cache projected to soon reach 24MB [3]), large translation lookaside buffers, out-of-order execu- tion, high speed buses, and support for a large number of pend- ing memory requests. These mechanisms result in large transistor counts leading to high leakage power and overall high power con- sumption under heavy load. In a modern processor, less than 20% of the transistor count is dedicated to the actual cores [13, 21]. Low power designs, on the other hand, focus on those processor features with low power operations. For example, the Atom proces- sor [14] includes an in-order pipeline that can execute two instruc- tions per cycle, a small L2 cache and power-efficient clock distri- bution. This results in a strongly reduced transistor count with low leakage power and limited power consumption at low load. Further, Atom design is focused on allowing quick and frequent transitions to a very low power state (e.g., 80 mW with less than 100 μs exit latency [14]). Proposals like FAWN [12] and Marlowe [5] explore these features to build arrays of low power servers that operate ef- ficiently for specific I/O bound workloads. In summary, we observe a dichotomy between low power and high performance system designs. Choosing the most appropriate design for an energy efficient datacenter is far from straightforward. First, datacenter workloads are diverse – some (e.g., I/O-bound map/reduce like) lend themselves rather easily to low power de- signs while others (e.g., transactions, encryption) depend on high performance and fast response times to satisfy stringent service- level agreements (SLAs). Second, the workload dynamics— including job arrival patterns and completion times—may reverse the conclusion of static workload analysis. Finally, the processor is just one contributor to the overall power consumption. Other system components such as the motherboard (e.g., I/O and mem- ory controllers), DRAM banks and power supplies contribute to a large fraction of the overall power consumption and tend not to be optimized for low power operation. Given these challenges, we propose a hybrid solution that mixes low power systems and high performance ones. In Section 2 we perform a preliminary evaluation to highlight the potential of hy- brid solutions. Then in Section 3 we layout the challenges of such solutions and explore the design options in this space by giving a quick overview of the spectrum of solutions. Finally, we conclude the paper with a summary of related work. 2. THE CASE FOR HYBRID AP- PROACHES As a first step, we are interested in comparing the performance of different systems under datacenter-like workloads. For this task, we consider a quad-core, dual-socket Xeon system and two low-