Visigoth Fault Tolerance Daniel Porto † , Jo˜ ao Leit˜ ao † , Cheng Li ‡ , Allen Clement ‡* , Aniket Kate \ , Flavio Junqueira ] and Rodrigo Rodrigues † † NOVA Univ. Lisbon / NOVA LINCS, ‡ MPI-SWS, \ MMCI/Saarland University, ] Microsoft Research Abstract We present a new technique for designing distributed pro- tocols for building reliable stateful services called Visigoth Fault Tolerance (VFT). VFT introduces the Visigoth model, which makes it possible to calibrate the timing assumptions of a system using a threshold of slow processes or mes- sages, and also to distinguish between non-malicious ar- bitrary faults and correlated attack scenarios. This enables solutions that leverage the characteristics of data center systems, namely their secure environment and predictable performance, in order to allow replicated systems to be more efficient with respect to the utilization of resources than those designed under asynchrony and Byzantine as- sumptions, while avoiding the need to make a system syn- chronous, or to restrict failure modes to silent crashes. We implemented a VFT protocol for a state machine replication library, and ran several benchmarks. Our evaluation shows that VFT has comparable performance to existing schemes and brings significant benefits in terms of the throughput per dollar, i.e., the server cost for sustaining a certain level of request execution. 1. Introduction Techniques have been proposed over the past few years to make the performance of both data center networks [48, 52] and data center systems [32] more predictable. Predictability is important because systems in data centers often comprise and depend on a number of networked servers and opera- tions require a subset of those servers to be contacted and to exchange messages. Without predictable performance, the quality of the provided service might fall short of the de- * currently working at Google. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. EUROSYS ’15, April 21-24, 2015, Bordeaux, France. Copyright c  2015 ACM 978-1-4503-3238-5/15/04. . . $15.00. http://dx.doi.org/10.1145/10.1145/2741948.2741979 manding requirements of users of online services [22], and even of offline services such as batch processing [51]. However, despite this trend of increasing predictability in performance within the data center, the design of repli- cation protocols for stateful services that run inside data centers is still making the same pessimistic assumptions re- garding timeliness that are commonly used for unpredictable environments like the Internet. For example, systems like Chubby [9], Spanner [14], Megastore [7], or ZooKeeper [25] use at its core the Paxos [30] consensus algorithm and vari- ants [27], which assume an asynchronous system, where all messages and processing events can be arbitrarily slow. A similar argument to the one made above regarding the pessimistic assumptions on timeliness can also be made re- garding non-crash faults. There is increasing evidence that machines and networks fail in unexpected ways that are not captured by the crash fault model, particularly at the scale of a data center, where the unlikely becomes commonplace [3– 5, 26]. While this is addressed by Byzantine Fault Toler- ance (BFT) techniques, BFT is unnecessarily conservative for data center environments. This is because BFT is de- signed to cope with coordinated malice, which is unlikely to happen within the security perimeter of the data center. (In fact, this excess of pessimism has been pointed out as one of the obstacles for the adoption of BFT in data center environments [45].) In this paper, we take the position that it is possible to take advantage of the fact that data centers are more pre- dictable and controllable than an open Internet environment, in order to make stateful services more resource efficient. Furthermore, this can be achieved without having to make assumptions that might be difficult to meet in practice, such as assuming a fully synchronous system where all machines and all messages meet tight deadlines, or that data corrup- tion never occurs. By resource efficient, we mean cutting the replication factors of systems like Paxos, which is an im- portant goal since ultimately this can lead to savings in both infrastructure and energy costs, which represent the vast ma- jority of the costs for operating a data center [24]. To demonstrate this, we present a new technique for de- signing distributed protocols for reliable stateful services called Visigoth Fault Tolerance (VFT). VFT introduces the