The Potential Influence of Workload Management Across Heterogeneous Server Systems on Datacenter Energy Use and Power Draw. D.H.Harryvan Certios BV Amsterdam, The Netherlands dirk.harryvan@certios.nl Giulio Urlini Advanced Systems Technology STMicroelectronics Agrate Brianza (MB), Italy giulio.urlini@st.com R. Chamberlane Nallatech Molex Cumbernauld, UK richard.chamberlain@molex.com O. Terzo Advanced Computing and Electromagnetics (ACE) Istituto Superiore Mario Boella (ISMB) Torino TO, Italy terzo@ismb.it Alberto SCionti Istituto Superiore Mario Boella (ISMB) Torino TO, Italy scionti@ismb.it Abstract—Energy efficiency is a key part of the European energy policies and 2020 climate targets. Project OPERA is working to create an energy aware workload manager for heterogeneous systems that will allow microservices to migrate between systems with differing instruction set architectures. The Energy savings potential of such technologies is enormous and is estimated at 47 TWh per year in Europe, 95% of the energy consumed by servers in Europe. The impact of such technologies on datacenter operations is profound. Significant and fast variations in power draw over time are expected, a fact that operators need to consider when retrofitting or designing new facilities. Keywords--workload management; heterogeneous server systems; project OPERA; energy savings I. INTRODUCTION Energy efficiency is a key part of the European energy policies and the 2020 and 2030 energy and climate targets include a specific part for energy efficiency [1]. Throughout the EU, policies and measures have been introduced to improve energy efficiency in every sector of the economy, including ICT and data centers [2]. For a number of years now, the datacenter industry has put tremendous effort in optimizing the Power Usage Effectiveness (PUE) as the main path to energy savings. From the definition of the PUE [3] it follows that:     =      The effect of best practices, such as the European Code of Conduct on Data Centre Energy Efficiency [2], has been published and a gradual improvement in the PUE, use has been reported [4]. This improvement has however, not led to a decrease in total energy use (see figure 1) and, given the reporting period, it is not expected that major improvements in the average PUE and total energy use are imminent. Other than decreasing the PUE, another option for lowering overall energy use is decreasing the energy use of the IT equipment. Since the peak in CPU clock frequencies in 2005, manufacturers have improved system performance in other directions such as developing multicores, dedicated accelerators and massive memory expansions. As a result, computational power has increased by orders of magnitude, while the overall power draw of these systems has dropped or stayed level. The total effect of these improvements is summed up in the so called “Koomey’s Law” [5] that states that it takes about 2.7 years for peak-output efficiency to double. Koomey does mention in this article that these 2.7 years is a marked slowdown of the improvements over the rate governing the years prior, which implies that new directions for improving efficiency need to be developed. An issue with the current computer systems is that in order to truly exploit their energy efficiency, these systems should be running near maximum workload all the time. Even in the most modern systems efficiency drops at low utilization. This was recognized by Jonathan Koomey in 2015 and his article mentioned earlier, introduced “typical use efficiency” as a way to describe these advances; not only in chip efficiency but also in power and workload management. II. PROJECT OPERA. The This paper reports on the advances in “energy aware workload management” across heterogeneous systems. This work is conducted under the flag of the “OPERA” project; a research project under funding by the European Union through the Horizon 2020 Research and Innovation Programme under the grant agreement No. 688386 [6] Workload management in itself is not a new concept, VMware Vmotion technology is a well-known example but as to date has been limited to homogeneous systems; the workload can be moved between nearly identical computers. The focus of OPERA is to broaden this to heterogeneous systems. Heterogeneity literally means “composed from dissimilar parts” and refers to the way that system manufacturers are currently pushing the boundary of performance. The combination of specialized processors such as GPU’s and crypto accelerators with more general-purpose