International Journal of Scientific & Engineering Research Volume 10, Issue 1, January-2019 1239 ISSN 2229-5518 IJSER © 2019 http://www.ijser.org An Attempt to Design and Implement Contiki and Cooja Regression Test Suites using Combinatorial Testing Abhinandan H Patil BITS Pilani, Goa Zuari Nagar, Goa Goa 919886406214 Abhinandan_patil_1414@yahoo.com Neena@goa.bits-pilani.ac.in Krishnan Rangarajan Dayanand Sagar College Bangalore Karnataka Krishnanr1234@gmail.com ABSTRACT Although several studies have been conducted on regression test selection, augmentation, prioritization, and pruning, regression test suite creation is performed in an ad hoc manner for most software systems by selecting a few crucial parameters and their combinations. Regression test suite creation can be rigorously approached by using the combinatorial testing (CT) method. Although this approach is advantageous, it has been used in only a few cases. This paper presents a practical approach for applying CT and augmenting the Contiki and Cooja base regression test suites with the National Institute of Standards and Technology tools. We discuss the results of using CT for the Contiki and Cooja regression test suites. In our study, we examined the inadequacies of existing regression test suites. We then incorporated additional tests to re-engineer the existing test suites by using the Automated Combinatorial Testing for Software tool. The test suite was written from scratch for the Cooja simulator of the Contiki operating system. In each case, we measured the coverage of the Contiki simulator. We observed a marginal increase in the simulator code coverage for the re-engineered test suite. However, a substantial increase was observed in the simulator coverage for the test suite written with the simulator acting as a system under test. In this study, we also examine the automation of test case generation for Contiki and Cooja. Key words: ACTS, CT, NIST 1. INTRODUCTION Software systems developed and maintained by teams must have a robust regression testing mechanism in place. This mechanism is required because the changes in the codebase must be tested, and the new code must be incorporated almost overnight. In most cases, a regression test suite is created in an adhoc manner by selecting a few crucial parameters and their combinations. This results in a small test suite that can be executed rapidly; however, the obtained test suite neglects crucial test cases and does not satisfy the testing requirements. In this study, we used the Automated Combinatorial Testing (CT) for Software (ACTS) tool of the National Institute of Standards and Technology (NIST) [1] to generate combinatorial test cases from the Contiki and Cooja regression test suites [2]. We used the code coverage as a criterion for measuring the testing adequacy. Contiki is an operating system, whereas Cooja is a simulator. Contiki and Cooja are primarily used in embedded devices [3]. They have a large code with several input parameters. We can build a practical and reliable regression test suite by augmenting an existing regression test suite through additional tests generated with a combinatorial approach. 2. BACKGROUND Contiki is a widely accepted Internet of Things (IoT) operating system and is suitable for memory- and resource-constrained devices. Contiki is an open-source software with a substantial user community. Contiki software includes Instant Contiki, which is a user-friendly environment for testing. By using VMWare, Instant Contiki can be launched in a desktop environment. The Ubuntu-based environment includes toolchain dependencies that help in making incremental changes to the operating system easy. Contiki can be built for various target platforms by tweaking the make file. Moreover, Contiki supports several hardware platforms. Instant Contiki is built in a Java Simulator tool called Cooja, which interacts with Contiki through the Java Native Interface. Cooja has a standard regression test suite in the regression test folder. The test suite comprises XML files with the csc extension. Cooja can understand these csc files. The XML files have information regarding the configuration and arrangement of the mote type as well as scenario- specific JavaScript embedded in them. Although Contiki supports various hardware platforms, the regression test suite does not reflect it. The test cases are concentrated around limited mote types. In a preliminary investigation, we noticed a scope for improving the test suite. Two additional test suites were planned. These additional test suites were named the re-engineered and Cooja test suites. Details regarding the test suites are provided in Table I. Table I. Test suites and their description Test suite System under test Additional comments Base test suite Mainly Contiki This can be found in a regression folder. This suite is referred to as “Test Suite A” in this paper. Re-engineered test suite Primarily Contiki This suite was created by selecting additional test cases from the ACTS tool. This test suite is referred to as “Test Suite B.” Cooja test suite Primarily Cooja simulator This test suite was created from scratch. This test suite is IJSER