1 Towards a High Performance Virtualized IaaS Deployment Andrew J. Younge 1, 2 , John Paul Walters 1 , Jinwoo Suh 1 , Dong-In D. Kang 1 , Youngsam Park 1 , Stephen P. Crago 1 , Geoffrey C. Fox 2 1 Information Sciences Institute, University of Southern California 3811 North Fairfax Drive, Suite 200, Arlington, VA 22203 U.S.A. 2 Pervasive Technology Institute, Indiana University 2729 E 10th St., Bloomington, IN 47408, U.S.A. Email corresponding author: ayounge@isi.edu Abstract—Scientific computing endeavors have created clus- ters, grids, and supercomputers as high performance computing (HPC) platforms and paradigms. These resources focus on peak performance and computing efficiency, thereby enabling scientific community to tackle non-trivial problems on massively parallel architectures. Meanwhile, efforts to leverage the economies of scale from data center operations and advances in virtualization technologies have created large scale Clouds. Such Infrastructure- as-a-Service (IaaS) deployments provide mechanisms for han- dling millions of user interactions concurrently or organizing, cataloging, and retrieving mountains of data in a way that allows users to specify a custom computing environment tailored to their needs. Combining concepts from both supercomputing and clouds will enable users to leverage the performance of HPC applications with the ease and availably in clouds, creating an ideal environment for many scientific computing applications. This work proposes building a heterogeneous, high performance IaaS using the OpenStack software concentrating on virtual- ization performance, heterogeneous hardware and GPUs, high speed interconnects, advanced scheduling and high performance storage to create a novel IaaS supercomputing system. I. I NTRODUCTION For many decades, the HPC community has led the march for the best possible computational performance, striving for absolute speed made which is possible through the applica- tion of Moore’s Law. This pursuit can be seen through the focus on cutting edge architectures, high-speed, low-latency interconnects, parallel languages, and dynamic libraries, all tuned to maximize computational efficiency. Performance has been the keystone of HPC since its conception, with many ways to evaluate performance. The Top500 supercomputer ranking system, which has been running for over 20 years, is a hallmark to performance’s importance in supercomputing. Many other benchmarks targeting the HPC realm exist to provide additional information about supercomputing impor- tance. This includes HPCC and SPEC benchmarking suites, as well as the new Graph500 list focusing on data intensive performance [1], [2]. Using these benchmarks to compare traditional supercomputers to IaaS deployments represent the key mechanism to evaluating the success of a HPC cloud. The goal of this work is not to identify the differences between supercomputers and clouds, but rather to illustrate the need for a cohesive architecture that blends the two distinct paradigms together. We propose the creation of a heteroge- neous, high-performance IaaS Cloud architecture leveraging the OpenStack project [3]. If such a system is successful, it will lower the barrier of entry for scientists and enable a new wave of scientific computation that is otherwise missed. Fur- thermore, such an architectural model will improve efficiency within data centers, increasing the potential for fine-grained computation, leading to reduced resource expenditures and/or increased resource utilization. II. STRATEGY To bring a heterogeneous, high performance cloud cyberin- frastructure to fruition, a few focal areas are needed to ensure the success of such an endeavour. Specifically, we outline 5 focal points; hypervisor performance, heterogeneous hardware, high speed interconnect support, advanced scheduling mech- anisms, and distributed storage; all wrapped in custom IaaS cloud architecture based on OpenStack. A. Virtualization Performance One of the biggest concerns surrounding the use of hyper- visors in production supercomputing resources is the perfor- mance overhead. While overhead was considerable in the past, recent hardware and hypervisor advancements have brought the overhead to near-native performance for many cases [4]. Furthermore, new methods for providing virtualized environ- ments have proven to provide near-native performance through chroot environments, particularly using LXC. B. Heterogeneous Hardware & GPUs While the supercomputing industry moves from petascale to exascale, systems are continually leveraging accelerator cards, typically general purpose graphical processing units (GPUs). These GPUs are able to provide 1-2 orders of mag- nitude increase in processing power compared to CPUs while only adding a linear increase in power usage. As such, the performance-to-watt ratio for GPUs are very high compared to CPUs, a factor that’s increasingly important as we move to multi-megawatt supercomputers and exascale. Historically, providing such add-on accelerators has not been possible in a virtualized environment until recently. Cur- rent work utilizes PCI passthrough functionality in hypervisors to provide such hardware directly to a VM. Through this, GPUs become available in Openstack, a cloud IaaS project. Currently nVidia Tesla GPUs are supported through LXC [5], with a more pronounced Xen integration is forthcoming. This will enable supercomputing-class hardware in the more desirable cloud environment.