Scalable I/O Virtualization via Self-Virtualizing Devices Himanshu Raj Ivan Ganev Karsten Schwan CERCS, College of Computing Georgia Institute of Technology Atlanta, GA 30332 {rhim,ganev,schwan}@cc.gatech.edu Jimi Xenidis IBM T.J. Watson Research Lab Yorktown, NY jimix@watson.ibm.com Abstract The virtualization of I/O devices is an integral part of system virtualization. This includes both virtualizing the physical devices and managing them across multiple guest virtual machines (VMs) or domains running on top of a virtual machine monitor (VMM) or hypervisor (HV). This paper presents the notion of self-virtualizing devices, where for higher end, ‘smart’ I/O devices, selected virtualization functionality is offloaded onto the devices themselves. In particular, a self-virtualizing device is aware of being used in a virtualized environment and implements the virtual device abstraction on the device itself. It also presents an interface to the HV to manage these virtual devices. Using this virtual device abstraction, a guest domain can interact with the physical device with minimal HV involvement. The outcomes are reduced I/O virtualization costs caused by the HV (and additional software components required to virtualize the device) and improved scalability for device interactions with guest domains. The prototype self-virtualizing device described in this paper leverages the multi-core nature of mod- ern computer architectures. It uses the IXP2400 network processor with multiple internal specialized communication cores to implement a self-virtualizing network interface attached to a x86 based host ma- chine. The device is used in conjunction with the Xen hypervisor and provides virtual network interfaces (VIFs) to the guest domains. Initial results demonstrate that the performance of such a self-virtualizing network interface exceeds that of network interfaces virtualized in software by the HV and its driver domain(s); for a single guest domain, ∼ 50% improvement in latency is obtained. Further, device per- formance scales well with increasing the number of guest domains, in part because the self-virtualizing network device reduces system resource utilization by the driver domain and the HV. Additional insights from the design and implementation of self-virtualizing devices concern the device-side functionality ap- propriate for effective virtualization and issues concerning the efficient interaction of HV- or domain-based activities with such devices, e.g. in future heterogenous multi-core systems. 1 Introduction Virtualization technologies have long been used for high-end server class systems to provide server consoli- dation for optimal resource usage, such as pSeries and zSeries machines from IBM. More recently, they are becoming an integral element of processor architectures that include lower end PowerPCs and x86-based machines [4]. The basis for virtualization are the hypervisors (HVs) or Virtual Machine Monitors (VMMs) running on such machines, which support the creation and execution of multiple guest domains on the same 1