A Survey on Edge Benchmarking Blesson Varghese a , Nan Wang b , David Bermbach c , Cheol-Ho Hong d , Eyal de Lara e , Weisong Shi f , and Christopher Stewart g a Queen’s University Belfast, UK; b Durham University, UK; c TU Berlin, Germany, and Einstein Center Digital Future, Germany; d Chung-Ang University, S. Korea; e University of Toronto, Canada; f Wayne State University, USA; g Ohio State University, USA This manuscript was compiled on September 8, 2021 Edge computing is the next Internet frontier that will leverage com- puting resources located near users, sensors, and data stores for delivering more responsive services. Thus, it is envisioned that a large-scale, geographically dispersed and resource-rich distributed system will emerge and become the backbone of the future Inter- net. However, given the loosely coupled nature of these complex systems, their operational conditions are expected to significantly change over time. In this context, the performance of these systems will need to be captured rapidly, referred to as benchmarking, for ap- plication deployment, resource orchestration, and adaptive decision- making. Edge benchmarking is a nascent research avenue that has started gaining momentum over the last five years. This article firstly examines articles published over the last three decades to trace the history of benchmarking from tightly coupled to loosely coupled sys- tems. Then it systematically classifies research to identify the sys- tem under test, techniques analyzed, quality metrics, and benchmark runtime in edge benchmarking. Edge computing | Edge benchmarking | System under test | Benchmark- ing techniques | Quality | Benchmark runtime 1. Introduction The computing landscape has significantly changed over the last three decades – loosely coupled and geographically dis- persed systems started replacing tightly coupled monolithic systems (1). One example is from two decades ago when computing resources that were distributed across numerous or- ganizations and continents were connected under the umbrella of grid computing. Grids then offered the unique capability of processing large datasets near the source without requiring to transfer data of a distributed workflow to a central system (2). Subsequently, computing became a utility offered remotely through the cloud (3). While the cloud is the current computing model for most Internet-based applications, it has been recognized as an un- tenable model for the future. This is because billions of devices and sensors are connected to the Internet, and data generated from these cannot be transferred and processed in geographi- cally distant cloud data centers without incurring delays (4). The next disruption in the computing landscape therefore is to further distribute infrastructure resources, and subsequently, application services to bring compute closer to the edge of the network near the data source (5). In this article, we use the term ‘edge computing’ to refer to the use of resources located at the edge of the network, such as routers and gateways or dedicated micro data centers, to either provide applications with acceleration by co-hosting services along with the cloud or by hosting them natively on edge resources (6). The inclusion of edge resources for computing creates a large-scale, geographically dispersed and resource-rich dis- tributed system that spans multiple technological domains and ownership boundaries. Such a complex system will be transient – the resources, their availability and characteristics change over time. For example, an edge resource previously available to an application may be unavailable due to a recent fault or the operating system of a target resource may have changed during maintenance (7, 8). In this context, it is essential to ad- dress the challenge of understanding the relative performance of applications by comparing target hardware platforms from different vendors due to diversity of hardware architectures and the impact on performance when system software level changes or new networking protocols are introduced (9). These are considered in edge benchmarking (10, 11). Benchmarking is the process of creating stress on a system while closely observing its response on a wide-range of quality metrics. Typically, synthetic or application-driven workloads are executed against a system under test, for example, vir- tual machines, storage systems, stream processing systems, or specific application components, while measuring quality be- haviors such as I/O throughput, data staleness, or end-to-end communication or computation latency. In contrast to alter- native approaches, such as predictive methods or simulation, insight into real system behavior is obtained by replicating the analyzed conditions of a production deployment (12). The focus of this article is three-fold. Firstly, it traces the history of the development of benchmarks over the last three decades for high-performance computing (HPC), grid, and cloud systems. Secondly, it catalogs and examines different edge benchmarks. Finally, the system under test, techniques analyzed, quality, and benchmark runtime that underpin edge benchmarking are reviewed. Figure 1 shows a histogram of the total number of research publications reviewed by this article between 1976 and 2020 under the categories: (i) books and book chapters, (ii) reports, including pre-print articles or white papers, and doctoral research thesis, (iii) conference or workshop papers, and (iv) journal or magazine articles. More than 85% of the articles reviewed have been published since 2010 and more than 65% of the articles since 2015. The remainder of this article is organized as follows. Sec- tion 2 provides a brief history of benchmarking. Section 3 catalogs different edge benchmarks. Section 4 presents a re- view of the system under test in edge benchmarking. Section 5 reviews the techniques analyzed in edge benchmarking. Sec- tion 6 highlights the quality aspect of edge benchmarking. Section 7 surveys the runtime execution environment and de- ployments in edge benchmarking. Section 8 suggests future directions for furthering research in edge benchmarking and concludes the paper. a Corresponding e-mail: b.varghese@qub.ac.uk | September 8, 2021 | 1–22 arXiv:2004.11725v1 [cs.DC] 24 Apr 2020