A practical evaluation of spectrum-based fault localization q Rui Abreu a, * , Peter Zoeteweij a , Rob Golsteijn b , Arjan J.C. van Gemund a a Delft University of Technology, The Netherlands b NXP Semiconductors, The Netherlands article info Article history: Available online 26 June 2009 Keywords: Test data analysis Software fault diagnosis Program spectra Real-time and embedded systems Consumer electronics abstract Spectrum-based fault localization (SFL) shortens the test–diagnose–repair cycle by reducing the debug- ging effort. As a light-weight automated diagnosis technique it can easily be integrated with existing test- ing schemes. Since SFL is based on discovering statistical coincidences between system failures and the activity of the different parts of a system, its diagnostic accuracy is inherently limited. Using a common benchmark consisting of the Siemens set and the space program, we investigate this diagnostic accuracy as a function of several parameters (such as quality and quantity of the program spectra collected during the execution of the system), some of which directly relate to test design. Our results indicate that the superior performance of a particular similarity coefficient, used to analyze the program spectra, is largely independent of test design. Furthermore, near-optimal diagnostic accuracy (exonerating over 80% of the blocks of code on average) is already obtained for low-quality error observations and limited numbers of test cases. In addition to establishing these results in the controlled environment of our benchmark set, we show that SFL can effectively be applied in the context of embedded software development in an industrial environment. Ó 2009 Elsevier Inc. All rights reserved. 1. Introduction Testing, debugging, and verification represent a major expendi- ture in the software development cycle (Hailpern and Santhanam, 2002), which is to a large extent due to the labor-intensive task of diagnosing the faults (bugs) that cause tests to fail. Because under typical market conditions, only those faults that affect the user most can be solved before the release deadline, the efficiency with which faults can be diagnosed and repaired directly influences software reliability. Automated diagnosis can help to improve this efficiency. Diagnosis techniques are complementary to testing in two ways. First, for tests designed to verify correct behavior, they gen- erate information on the root cause of test failures, focusing the subsequent tests that are required to expose this root cause. Sec- ond, for tests designed to expose specific potential root causes, the extra information generated by diagnosis techniques can help to further reduce the set of remaining possible explanations. Given its incremental nature (i.e., taking into account the results of an entire sequence of tests), automated diagnosis alleviates much of the work of selecting tests in the latter category, and can hence have a profound impact on the test–diagnose–repair cycle. An important part of diagnosis and repair consist in localizing faults, and several tools for automated debugging and systems diagnosis implement spectrum-based fault localization (SFL), an approach to diagnosis based on an analysis of the differences in program spectra (Harrold et al., 2000; Reps et al., 2000) for passed and failed runs. Passed runs are executions of a program that com- pleted correctly, whereas failed runs are executions in which an er- ror was detected. A program spectrum is an execution profile that indicates which parts of a program are active during a run. Spec- trum-based fault localization entails identifying the part of the program whose activity correlates most with the detection of er- rors. Examples of tools that implement this approach are Pinpoint (Chen et al., 2002), which focuses on large, dynamic on-line trans- action processing systems (Jones et al., 2002) whose implementa- tion focuses on the analysis of C programs, and AMPLE (Dallmeier et al., 2005), which focuses on object-oriented software (see Sec- tion 9 for a discussion). Spectrum-based fault localization does not rely on a model of the system under investigation. It can easily be integrated with existing testing procedures, and because of the relatively small overhead with respect to CPU time and memory requirements, it lends itself well for application within resource-constrained envi- ronments (Zoeteweij et al., 2007). However, the efficiency of SFL comes at the cost of a limited diagnostic accuracy. As an indication, 0164-1212/$ - see front matter Ó 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.jss.2009.06.035 q This work has been carried out as part of the TRADER project under the responsibility of the Embedded Systems Institute. This project is partially supported by the Netherlands Ministry of Economic Affairs under the BSIK03021 program. * Corresponding author. E-mail addresses: r.f.abreu@tudelft.nl (R. Abreu), p.zoeteweij@tudelft.nl (P. Zoeteweij), rob.golsteijn@nxp.com (R. Golsteijn), a.j.c.vangemund@tudelft.nl (A.J.C. van Gemund). The Journal of Systems and Software 82 (2009) 1780–1792 Contents lists available at ScienceDirect The Journal of Systems and Software journal homepage: www.elsevier.com/locate/jss