Building blocks for IoT testing - a benchmark of IoT apps and a functional testing framework Rares , Cristea rares.cristea@unibuc.ro University of Bucharest Bucures , ti, Romania Mihail Feraru mihail.feraru@s.unibuc.ro University of Bucharest Bucures , ti, Romania Ciprian Paduraru ciprian.paduraru@fmi.unibuc.ro University of Bucharest Bucures , ti, Romania ABSTRACT IoT security is a topic that ofers numerous opportunities for im- provement and development. In this paper, we frst present a set of open-source mock IoT applications along with the necessary infras- tructure specifcally designed to emulate a real IoT system. With our app set, users can add their own applications, automation rules, and communication fows with little technical efort, and test difer- ent scenarios to reproduce bugs that are not specifc to the use ofa single device. Second, we describe a functional testing framework for the IoT that is inspired by behavior-driven development (BDD), a testing methodology that serves as a proof-of-concept for how the application set can be used in diferent test scenarios. The appli- cation set and the functional testing framework are independent of each other. Our goal is to help IoT developers and testers fnd new testing techniques and benchmarking them in a reproducible, com- parable, and less biased environment. We believe that they form the basis for a better understanding of how to test systems composed of heterogeneous devices to fnd issues and vulnerabilities that arise mainly from their interaction and data persistence management. ACM Reference Format: Rares , Cristea, Mihail Feraru, and Ciprian Paduraru. 2022. Building blocks for IoT testing - a benchmark of IoT apps and a functional testing framework. In 4th International Workshop on Software Engineering Research and Practice for the IoT (SERP4IoT’22), May 19, 2022, Pittsburgh, PA, USA. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3528227.3528568 1 INTRODUCTION The Internet of Things (IoT) is an ever-growing feld today. Ex- amples such as smart cities connecting transportation networks, intelligent car systems, and smart home systems are more present in our lives than ever. Sensors, actuators, end-user applications, gateways, and servers are interconnected through various commu- nication protocols and methods without us even realising it. It is common for an IoT system to be composed of devices from dif- ferent vendors, resulting in a unique combination of technologies that interact in unpredictable ways. However, as one would expect, the rapid development and deployment cycles lead to major chal- lenges, including security and privacy issues. Problems can occur Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a fee. Request permissions from permissions@acm.org. SERP4IoT’22, May 19, 2022, Pittsburgh, PA, USA © 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-9332-4/22/05. . . $15.00 https://doi.org/10.1145/3528227.3528568 at multiple levels, from single isolated applications to networked application streams exposed to various risks, such as Distributed Denial-of-Service (DDoS) [1] , identity management and user secu- rity issues [20], etc. After reviewing the literature and fnding that there is a lack of common ground for evaluating and researching testing approaches, we add the following contributions in this paper from the authors’ knowledge: • The frst open-source set of IoT-specifc software applica- tions that serves as a benchmark for the community to test diferent testing methods and compare them efciently by providing a reproducible environment, a less biased com- parison of results, and a simpler process for evaluating the efectiveness of a particular tool or technique. In addition to this suite of applications, we also provide tools to automate the development process and communication infrastructure so that users can focus on the methods and algorithms rather than the technical work. We hope that facilitating and stan- dardizing these processes will increase interest in and accel- erate the advancement of testing methods in the IoT space. • The frst open source framework that serves as a basic me- thod for functional testing in the IoT domain. We consider that the proposed system can be easily extended by the user, with the possibility of adding their own set of applications and customized communication fows between them. We also provide the tools needed to automate issue detection. In our framework, the concept of issue can be understood as a template. Examples include exploit detection in the security domain, performance issues, or classic source code bugs. The user of the framework can be either a software developer or a quality assurance professional (tester). As a side story, the applications in our application set were initially developed as part of an undergraduate course in Software Engineer- ing at the University of Bucharest. The goal of the course is to teach students the general practices of software engineering in the feld of IoT. The students have to defne a free-form software development project, evaluating the topics presented in the lectures of the course: Application Analysis, Architecture, Security, Agile Practices, etc. In the 2020-21 edition of the course, students were required to create a software project that met the criteria of vision for our application set. As the literature suggests, providing students with a real-world use case for their work projects motivates them and leads to a better software development process and fnal product [15]. Since the applications are developed by multiple independent teams, they can better represent the IoT system scenario: Devices from diferent vendors are integrated into complex systems that interact in unpredictable ways and present new testing challenges. In the following Section, we review the existing literature on testing 25 2022 IEEE/ACM 4th International Workshop on Software Engineering Research and Practices for the IoT (SERP4IoT)