On generating mutants for AspectJ programs Fadi Wedyan a , Sudipto Ghosh b,⇑ a Department of Software Engineering, Hashemite University, Zarka, Jordan b Department of Computer Science, Colorado State University, Fort Collins, CO, USA article info Article history: Available online xxxx Keywords: AspectJ Aspect-oriented programming Mutation testing Fault models Test generation High order mutation abstract Context: Mutation analysis has been widely used in research studies to evaluate the effectiveness of test suites and testing techniques. Faulty versions (i.e., mutants) of a program are generated such that each mutant contains one seeded fault. The mutation score provides a measure of effectiveness. Objective: We study three problems with the use of mutation analysis for testing AspectJ programs:  The manual identification and removal of equivalent mutants is difficult and time consuming. We cal- culate the percentage of equivalent mutants generated for benchmark AspectJ programs using avail- able mutation tools.  The generated mutants need to cover the various fault types described in the literature on fault mod- els for AspectJ programs. We measure the distribution of the mutants generated using available muta- tion tools with respect to the AspectJ fault types.  We measure the difficulty of killing the generated mutants. We propose the use of simple analysis of the subject programs to prevent the generation of some equiv- alent mutants. Method: We revised existing AspectJ fault models and presented a fault model that removes the prob- lems in existing fault models, such as overlapping between fault types and missing fault types. We also defined three new fault types that occur due to incorrect data-flow interactions occurring in AspectJ pro- grams. We used three mutation tools: AjMutator, Proteum/AJ, and MuJava on three AspectJ programs. To measure the difficulty of killing the mutants created using a mutation operator, we compared the average number of the mutants killed by 10 test suites that satisfy block coverage criterion. Results: A high percentage of the mutants are equivalent. The mutation tools do not cover all the fault types. Only 4 out of 27 operators generated mutants that were easy to kill. Conclusions: Our analysis approach removed about 80% of the equivalent mutants. Higher order mutation is needed to cover all the fault types. Ó 2011 Elsevier B.V. All rights reserved. 1. Introduction Experimental studies in software testing require the evaluation of a testing approach in terms of its ability to detect faults in sub- ject programs containing known faults. However, researchers often have a problem in finding subject programs that meet their re- search requirements and also contain real faults. Even when such programs are available, the number of real faults is often not large enough to allow achieving statistically significant results [3]. Therefore, researchers typically seed faults in the subject pro- grams, either manually or with mutation operators. The latter ap- proach has several advantages. A large number of faulty versions can be generated, in a systematic and easily replicatable manner [26]. Moreover, Andrews et al. [3,4] provided evidence that faults generated with mutation operators are similar to real faults in evaluating test effectiveness, while hand-seeded faults are harder to detect than real faults. One major problem with mutation testing is that mutation operators generate a large number of equivalent mutants (i.e., faults produced by the mutants do not propagate to produce a dif- ferent output). These mutants need to be identified and eliminated from the analysis. In a recent study, Schuler and Zeller [30] showed that, for Java programs, about 45% of the undetected mutants turned out to be equivalent. They reported that it took about 15 min to assess one single mutant. Budd and Angluin [8] proved that detecting equivalent mutants is an undecidable problem. Therefore, many researchers have proposed techniques for partially detecting equivalent mutants. These techniques include using compiler optimization [6], constraint-based testing [28], program slicing [18,19,33], and co-evolutionary approaches [1]. For AspectJ programs, few studies have been performed that involve seeding faults using mutation operators. We can view aspect-oriented mutation with Aspectj as a superset of 0950-5849/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.infsof.2011.12.001 ⇑ Corresponding author. Tel.: +1 970 491 4608; fax: +1 970 491 2466. E-mail address: ghosh@cs.colostate.edu (S. Ghosh). Information and Software Technology xxx (2011) xxx–xxx Contents lists available at SciVerse ScienceDirect Information and Software Technology journal homepage: www.elsevier.com/locate/infsof Please cite this article in press as: F. Wedyan, S. Ghosh, On generating mutants for AspectJ programs, Inform. Softw. Technol. (2011), doi:10.1016/ j.infsof.2011.12.001