System-Level Virtualization for High Performance Computing * Geoffroy Vall´ ee Oak Ridge National Laboratory Oak Ridge, TN 37830, USA valleegr@ornl.gov Thomas Naughton Oak Ridge National Laboratory Oak Ridge, TN 37830, USA naughtont@ornl.gov Christian Engelmann Oak Ridge National Laboratory Oak Ridge, TN 37830, USA engelmannc@ornl.gov Hong Ong Oak Ridge National Laboratory Oak Ridge, TN 37830, USA hongong@ornl.gov Stephen L. Scott Oak Ridge National Laboratory Oak Ridge, TN 37830, USA scottsl@ornl.gov Abstract System-level virtualization has been a research topic since the 70’s but regained popularity during the past few years because of the availability of efficient solution such as Xen and the implementation of hardware support in com- modity processors (e.g. Intel-VT, AMD-V). However, a majority of system-level virtualization projects is guided by the server consolidation market. As a result, current virtualization solutions appear to not be suit- able for high performance computing (HPC) which is typi- cally based on large-scale systems. On another hand there is significant interest in exploiting virtual machines (VMs) within HPC for a number of other reasons. By virtualiz- ing the machine, one is able to run a variety of operating systems and environments as needed by the applications. Virtualization allows users to isolate workloads, improving security and reliability. It is also possible to support non- native environments and/or legacy operating environments through virtualization. In addition, it is possible to balance work loads, use migration techniques to relocate applica- tions from failing machines, and isolate fault systems for repair. This document presents the challenges for the implemen- tation of a system-level virtualization solution for HPC. It also presents a brief survey of the different approaches and * ORNL’s research sponsored by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory (ORNL), man- aged by UT-Battelle, LLC for the U. S. Department of Energy under Con- tract No. DE-AC05-00OR22725. techniques to address these challenges. 1 Introduction Today several operating systems (OS) are used for high- performance computing (HPC): Linux on clusters, CNK on BlueGene/L [21], and Catamount on Cray [11, 12]. This variety creates a gap between the initial application devel- opment and ultimate execution platforms. For example, a developer’s workstation may be used with a modest sized cluster for initial development followed by a porting phase to take the code to the HPC platform. That gap introduces an additional cost every time users want to execute an appli- cation on a new HPC platform. Also, as HPC systems grow in size and promise greater performance, the rate of failure increases stealing a portion of the increased performance and thus impact an application’s time to solution. One solution to address these issues is to use system- level virtualization. System-level virtualization creates an abstraction of the hardware and executes one or several vir- tual machines (VMs) on top of this virtualized hardware; in some instances, the virtual machine may also directly ac- cess the hardware for performance purposes. Virtualization solutions are today based on the concept of a Virtual Ma- chine Monitor (VMM), also called a hypervisor. The VMM is responsible for the hardware virtualization and execution of VMs on top of the virtualized hardware. The VMM is typically a small operating system that does not include hardware drivers. To access physical resources, the VMM 1