Automated and Eficient Test-Generation for Grid-Based Multiagent Systems Comparing Random Input Filtering versus Constraint Solving SINA ENTEKHABI and WOJCIECH MOSTOWSKI, Halmstad University, Sweden MOHAMMAD REZA MOUSAVI, King’s College London, UK Automatic generation of random test inputs is an approach that can alleviate the challenges of manual test case design. However, random test cases may be inefective in fault detection and increase testing cost, especially in systems where test execution is resource- and time-consuming. To remedy this, the domain knowledge of test engineers can be exploited to select potentially efective test cases. To this end, test selection constraints suggested by domain experts can be utilized either for iltering randomly generated test inputs or for direct generation of inputs using constraint solvers. In this paper, we propose a domain speciic language (DSL) for formalizing locality-based test selection constraints of autonomous agents and discuss the impact of test selection ilters, speciied in our DSL, on randomly generated test cases. We study and compare the performance of iltering and constraint solving approaches in generating selective test cases for diferent test scenario parameters and discuss the role of these parameters in test generation performance. Through our study, we provide criteria for suitability of the random data iltering approach versus the constraint solving one under the varying size and complexity of our testing problem. We formulate the corresponding research questions and answer them by designing and conducting experiments using QuickCheck for random test data generation with iltering and Z3 for constraint solving. Our observations and statistical analysis indicate that applying ilters can signiicantly improve test eiciency of randomly generated test cases. Furthermore, we observe that test scenario parameters afect the performance of the iltering and constraint solving approaches diferently. In particular, our results indicate that the two approaches have complementary strengths: random generation and iltering works best for large agent numbers and long paths, while its performance degrades in the larger grid sizes and more strict constraints. On the contrary, constraint solving has a robust performance for large grid sizes and strict constraints, while its performance degrades with more agents and long paths. CCS Concepts: · Software and its engineering → Software testing and debugging. Additional Key Words and Phrases: Test Input Generation, Domain Speciic Languages, Test Selection, Autonomous Agents, Multiagent Systems, Grid-based systems, Constraint Solving, Test Input Filtering 1 INTRODUCTION Testing typically accounts for more than half of the software development costs [35]. Test automation, e.g., using Model-Based Testing (MBT) [27] or Property-Based Testing (PBT) [10], mitigates this problem by generating tests at low additional cost once a model or a suitable property speciication is in place. However, for complex systems and speciic application areas, several problems remain. In particular, in autonomous and AI-enabled systems, the input domain is a huge multi-dimensional data space and it is not always clear how an efective sampling can be made. Second, test execution can be very time- and resource-intensive, even if it is limited to a simulation environment, let alone in the hardware- and vehicle-in-the-loop settings. Finally, for autonomous systems it is Authors’ addresses: Sina Entekhabi, sina.entekhabi@hh.se; Wojciech Mostowski, wojciech.mostowski@hh.se, Halmstad University, Halmstad, Sweden; Mohammad Reza Mousavi, King’s College London, London, UK, mohammad.mousavi@kcl.ac.uk. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proit or commercial advantage and that copies bear this notice and the full citation on the irst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). © 2023 Copyright held by the owner/author(s). 1049-331X/2023/9-ART https://doi.org/10.1145/3624736 ACM Trans. Softw. Eng. Methodol.