Advantages of mutation in passive testing: An empirical study. ∗ César Andrés, Mercedes G. Merayo, Carlos Molinero Deptartamento Sistemas Informáticos y Computación Universidad Complutense de Madrid E-28040 Madrid. Spain. {c.andres,mgmerayo,molinero}@fdi.ucm.es Abstract This paper presents an empirical study of the mutation techniques used by the tool PASTE. This tool allows the automation of our passive testing methodology for systems that present stochastic-time information. In our proposal, invariants express the fact that each time the implementa- tion under test performs a given sequence of actions, then it must exhibit a behavior according to the probability dis- tribution functions reflected in the invariant. We briefly re- view the theoretical framework of our methodology and the main features of our tool. Next, we present in detail the Mutants module that provides us with a functionality to test the effectiveness of invariants for detecting errors. Finally, we present a study of the results obtained from the performed experiments. 1 Introduction Formal testing techniques [6, 18, 16, 24] provide sys- tematic procedures to check implementations with respect to a specification in such a way that the coverage of criti- cal parts/aspects of the system under test is increased. Ini- tially, these techniques focused on the functional behavior of systems, such as determining whether the tested system can, on the one hand, perform certain actions and, on the other hand, does not perform some unexpected ones. Never- theless, there exist many systems where non-functional as- pects, such as the probability of an event to happen, the time that it takes to perform a certain action, or the time when a certain action happens, make the difference between correct and incorrect behaviors. During the last two decades there has been a lot of interest in extending formal techniques to cope with time. Even though there exist several propos- als for timed testing [19, 10, 14, 25, 11, 22, 20], most of ∗ Research supported by the Spanish MEC project WEST/FAST (TIN2006-15578-C02-01). them specialize in active testing, assuming that the tester is allowed to interact directly with the system. However, sometimes testers cannot interact with the system. In those situations, testing methodologies make use of monitoring techniques, that extract traces from the implementation that can be analyzed in order to detect errors. This paradigm is called passive testing. There are several proposals for formal passive testing [4, 5, 17], but they focuses on check- ing wether the systems satisfy functional properties, such as “after the input i the systems always emits the output o”. Our work is based on [4, 5]. In this approach, a set of properties, called invariants, are checked against the traces observed from the implementation to test their correctness. We extended this idea in [2] by adding stochastic time re- quirements in the specification and in the invariants. There, we present algorithms to verify the time-correctness of the invariants with respect to the specification and to assess the time-correctness of the recorder traces with respect to the invariants. We present a formalism based on finite state machines, allowing to take into account temporal aspects. The time is represented by means of probability distribu- tion functions, it allows that instead of having expressions such as “the action o takes t time units to be performed” we will have expressions such as “with probability p the action o will be performed before t time units”. We have developed a tool, called PASTE, where the al- gorithms proposed for checking the correctness of the in- variants and the recorder traces have been implemented. In addition to the automation of our methodology, the tool pro- vides the tester with a functionality that allows to check the effectiveness of invariants to detect errors, by apply- ing them to simulated traces. In order to generate these traces we have applied a methodology based on mutation testing [23, 12, 26, 21, 13]. Originally, mutation testing was applied to code [15, 7] but some work has looked at specification mutation [8]. Here, the specification is mu- tated and for each mutant a test is derived that distinguishes the behaviours of the mutated and original specifications. Most work on mutating specifications has used either finite IEEE International Conference on Software Testing Verification and Validation Workshops 978-0-7695-3671-2/09 $25.00 © 2009 IEEE DOI 10.1109/ICSTW.2009.33 230