Benchmarking the Performance of Hypervisors on Different Workloads Devi Prasad Bhukya, Carlos Gonçalves, Diogo Gomes, Rui L. Aguiar Instituto de Telecomunicações, Universidade de Aveiro, Aveiro, Portugal bdeviprasad@av.it.pt, cgoncalves@av.it.pt, dgomes@av.it.pt, ruilaa@ua.pt Abstract—Many organizations rely on a heterogeneous set of applications in virtual environment to deliver critical services to their customers. Different workloads utilize system resources at different levels. Depending on the resource utilization pattern some workloads may be better suited for hosting on a virtual platform. This paper discusses a framework for benchmarking the performance of Oracle database workloads, such as Online Analytical Processing (OLAP), Online Transaction Processing (OLTP), Web load and Email on two hypervisors, namely Xen and VMware. Further, Design of Experiments (DOE) is used to identify the significance of input parameters, and their overall effect on two hypervisors, which provides a quantitative and qualitative comparative analysis to customers with high degree of accuracy to choose the right hypervisor for their workload in datacenters. Keywords—Virtualization; DOE; Full Factorial Design; Main Effect; Interaction Effect I. INTRODUCTION Virtualization [1][2][3] is a technology where user can run more than one operating systems on a single system side by side. Initially, computer hardware was designed to run a single operating system at a time with a single application. This leaves most machines vastly underutilized. Virtualization lets user to create more than one Virtual Machine (VM) on a single physical machine and run different operating systems in different VMs with multiple applications on the same physical computer. Each VM shares the resources of that one physical computer across multiple environments under the monitoring of a Virtual Machine Monitor (VMM) or Hypervisor [4]. Virtualization works by inserting a thin layer of software called hypervisor directly on the computer hardware or on a host operating system. The virtualization architecture [5] we are using to do this experiment is by directly inserting the hypervisor on the hardware, which is called Bare-Metal architecture. Virtualization creates a virtual version of an operating system, a server, a storage device or network resources. The areas where virtualization is used are mainly network virtualization, storage virtualization and server virtualization. Virtualization is the best option available today for maximum utilization of the system resources by sharing of application and database. This helps to reduce the number of servers, hardware devices in the data center, which not only reduces the infrastructure and maintenance costs but also, reduces power consumption. The major benefit of using a Bare-Metal architecture is the overhead of a layer can be avoided; smaller footprint of the underlying Operating System (OS) uses considerably less system resources thereby granting the hypervisor and its VMs access to more Central Processing Unit (CPU) cycles, available memory and storage space on the hardware platform. There are different types of virtualization, for different approaches. User has a choice to select any type of virtualization depending on his application/workload need. The main categories are Storage virtualization, Hardware virtualization, Network virtualization and Server virtualization. There are three types of virtualization techniques [6] that are mainly used i.e., Full Virtualization, Para Virtualization and Hardware Emulation. Different workloads utilize hypervisor resources at different levels and depending on the resource utilization pattern some workloads may be better suited for hosting for particular hypervisor. This study is intended to compare how different Oracle workloads, such as Online Analytical Processing (OLAP), Online Transaction Processing (OLTP), Web Load and Email applications perform on different hypervisor environments. The rest of the work is organized as follows: Section 2 discusses about state of the art. Section 3 explains experimental procedure. Section 4 presents benchmarking analysis. Section 5 discusses about related work and Section 6 presents the conclusions. II. STATE OF THE ART Actually, performance of any system depends on various system and application factors. Higher performance is achieved in any system by tuning its individual system factors. Optimal values for each system tunable factor should be obtained by conducting several experimental runs and it takes a long time and blocks valuable resources such as cost, manpower and time. In traditional approach, performance benchmarking analysts are not aware of the experimental designs and analysis techniques often reaching misleading conclusions due to the following mistakes, such as: variations caused by experimental errors are not taken into account; important system parameters are not controlled; effects of different factors are not isolated; simple One-Factor-At-a- Time(OFAT) [22]designs are used; interactions among various factors are ignored; and too many experiments are conducted. In addition, traditional performance tuning and benchmarking of hypervisor systems continues to be a tedious and time-consuming job with respect to any workloads. Since the features of upcoming hypervisor products are so complex, benchmarking needs in depth knowledge of the product and its domain. In the real world, hypervisor customers usually comes with a benchmarking requirement for their product with their competitors to software service based company and they will always go for a cost effective way of benchmarking. They also identify systems having many parameters that require careful hand tuning to get good performance. 622 Copyright (c) IARIA, 2014. ISBN: 978-1-61208-367-4 ICSEA 2014 : The Ninth International Conference on Software Engineering Advances