Elastic Consistent Hashing for Distributed Storage Systems Wei Xie and Yong Chen Department of Computer Science, Texas Tech University, Lubbock, TX 79413 Abstract—Elastic distributed storage systems have been in- creasingly studied in recent years because power consumption has become a major problem in data centers. Much progress has been made in improving the agility of resizing small- and large-scale distributed storage systems. However, most of these studies focus on metadata based distributed storage systems. On the other hand, emerging consistent hashing based distributed storage systems are considered to allow better scalability and are highly attractive. We identify challenges in achieving elasticity in consistent hashing based distributed storage. These challenges cannot be easily solved by techniques used in current studies. In this paper, we propose an elastic consistent hashing based distributed storage to solve two problems. First, in order to allow a distributed storage to resize quickly, we modify the data placement algorithm using a primary server design and achieve an equal-work data layout. Second, we propose a selective data re-integration technique to reduce the performance impact when resizing a cluster. Our experimental and trace analysis results confirm that our proposed elastic consistent hashing works effectively and allows significantly better elasticity. I. I NTRODUCTION The growing power consumption is a critical issue for data centers. It adds substantial cost to an organization and not to mention the carbon footprint. Elastic distributed storage sys- tems have gained increasing attention in recent years in order to alleviate this problem [1]. Such storage systems attempt to achieve power-proportionality by making power consumption proportional to the dynamic system load leveraging the periods with light load. Most existing studies [2]–[5] consider resizing the storage cluster as the fundamental technique to achieve power propor- tionality. When the system load decreases, some of the storage servers in the cluster are powered off. When the load increases, these servers are powered back on. To achieve better elasticity by resizing the storage cluster, several techniques have been introduced in recent studies. For instance, a primary server design is introduced in Sierra [2] and has been used in several other studies too [3], [5]. It ensures that a full data copy is maintained in a small subset of servers called primary servers. With primary server design, no clean-up work is needed when powering down several storage servers. To achieve better performance, the data layout needs to be modified [3] for performance proportionality along with power proportionality. In addition, since the small number of primary servers limits the write performance, several recent studies [3], [5] propose to dynamically change the number of primary servers to balance the write performance and elasticity. Existing efforts have demonstrated the importance and effectiveness of provisioning elasticity in distributed storage systems. However, these efforts and methodologies focus on conventional distributed storage systems with metadata servers, for example the Hadoop Distributed File System (HDFS) [6]. Consistent hashing based distributed storage systems are emerging in data centers recently due to significantly better scalability potential and easy manageability [7]–[9]. Compared to distributed storage systems with metadata servers, they do not have a metadata service to manage the mapping from data to storage servers. Instead, they use a deterministic hash func- tion to determine the placement of data items. However, these different characteristics of consistent hashing based distributed storage make it challenging to achieve elasticity and power- proportionality. In this study, we introduce a series of techniques for en- abling the power-proportionality for consistent hashing based distributed storage systems. The design of the elastic consistent hashing is inspired by existing techniques that are proven effective in prior studies. We introduce primary server data placement and equal-work data layout to make it possible to scale down to very few active servers with small data move- ment. We propose a selective data re-integration technique to alleviate the performance degradation when a cluster resizes. The proposed techniques are implemented on Sheepdog stor- age system and evaluated with a storage cluster testbed. We also conduct analysis based on real-world traces and compare our proposed techniques against the original system. We find that the proposed elastic consistent hashing techniques are able to achieve both better elasticity and performance, and saves significantly more machine hours (which means power consumption). The contribution of this study includes: • To the best of our knowledge, this is the first study to systematically investigate the elasticity in consistent hashing based storage systems. • We introduce an elastic consistent hashing that not only adapts existing elasticity techniques to consistent hashing but also introduces new selective data re- integration techniques to alleviate the performance impact of data migration. • We demonstrate the effectiveness of the elastic consis- tent hashing techniques based on a Sheepdog storage system. Our solutions significantly outperform the original consistent hashing based Sheepdog system. • We analyze real-world traces and confirm that elastic consistent hashing is able to substantially reduce the power consumption in many cases.