mWarp: Accelerating Intra-Host Live Container Migration via Memory Warping Piush K Sinha, Spoorti S Doddamani, Hui Lu, and Kartik Gopalan State University of New York (SUNY) at Binghamton {psinha1, sdoddam1, huilu, kartik}@binghamton.edu Abstract—Live container migration allows containers to roam from one server to another to achieve agility goals like load bal- ancing, tackling machine failures, scaling in/out and reallocating resources. However, migrating a container is also costly mainly due to memory state migration — a large number of memory pages need to be copied from the source server to the destination server. In this paper, we propose a fast and live container migration approach, mWarp, in an intra-host scenario, where both the source and destination virtual machine (VM) servers reside on the same physical host. Instead of copying a container’s memory, mWarp relocates the ownership of the container’s physical memory pages from the source VM to the destination VM with a highly-efficient memory remapping mechanism. As relocation of memory ownership is light-weight, mWarp leads to fast and live container migration with less service disruption to applications running in containers being migrated. We implement mWarp upon a well-known live container migration tool (CRIU) with key kernel/hypervisor-level support. The evaluation with both micro benchmarks and real-world applications shows that mWarp greatly reduces the total container migration time and downtime (e.g., by an order of magnitude) with significantly improved application-level performance (e.g., by 20%). Index Terms—Virtualization, Container, Live Migration I. I NTRODUCTION As an alternative to virtual machine (VM) based virtual- ization (e.g., KVM, VMware, Xen), containers relying on process-based virtualization offer a much flexible way in deploying and executing applications. With containers, a bunch of new use cases have been enabled in clouds such as (micro- )services orchestration, management and just-in-time deploy- ment [1]–[5]. For better isolation and security in multi-tenant public clouds [6]–[10], containers are commonly encapsulated in VMs while running. For example, containerized applications are orchestrated and managed by Google Kubernetes Engine upon a group of Google Compute Engine instances (i.e., VMs) [3]. Like VMs, in-cloud containers need the capability to mi- grate from one VM to another to achieve agility goals like balancing load, escaping from hardware failures, scaling in/out demands for resources, etc. A “cold” container migration pro- cedure [11] usually involves three main steps: (1) suspending the container on the source VM; (2) copying the state of the container from the source VM to the destination VM; and (3) restoring the container on the destination VM with the same state as that on the source VM. In practice, a live con- tainer migration is more appealing which keeps the container running during migration — without disrupting the services of applications running within. For example, precopy-based live container migration [12] allows the migrated container to keep running on the source side while its memory gets transferred to the destination in an iterative manner (i.e., only the dirtied memory is transferred in each iteration), until the dirtied portion is small enough when the container is paused and its remaining state is copied (less migration downtime). Though the above approaches [12], [26] reduce the migra- tion downtime, they in turn lengthen the total migration time (i.e., the time between the start and the end of the migration) due to either iterative memory copying [12] or on-demand memory pages fetching [26]. During the total migration time, the performance of applications running in containers could be negatively impacted, for example, by costly page-fault-based dirty memory tracking or massive migration network traffic. In this paper, we present mWarp, a fast and live container migration approach targeting a common intra-host migration scenario in public clouds: When performing container migra- tion, it is preferable to choose/provision a destination VM on the same physical host (as long as the underlying host remains available with sufficient resources), as the intra-host migration can leverage local memory bandwidth for fast state transferring and avoid costly inter-host network communication. Such intra-host container migration is particularly applicable for a VM that needs to be temporarily shut down for maintenance, upgrade, and recovering from failures during which its hosted processes/containers must be migrated. The intra-host live container migration makes the key idea of mWarp feasible: As the source and destination VMs reside on the same physical host sharing the same memory, mWarp completely avoids memory copying by relocating the own- ership of a container’s memory pages from the source VM to the destination VM — which we call memory warping. To realize a highly-efficient memory relocation mechanism, mWarp involves a new page table, mWarp table, residing in the host kernel. During a container’s migration, the mWarp table keeps track of the mappings from the container’s address space to its host physical address on the source VM, and exposes such mapping information for reconstructing the containers memory space on the destination VM. As mWarp’s memory relocation is super fast in comparison with traditional memory copying, mWarp leads to sub-second intra-host live container migration, regardless of the size of the containers being migrated. In contrast, the memory copying based intra-host container migration can take several or tens of seconds.