Practical Fault Localization for Dynamic Web Applications Shay Artzi Julian Dolby Frank Tip Marco Pistoia IBM T.J. Watson Research Center P.O. Box 704, Yorktown Heights, NY 10598, USA {artzi,dolby,ftip,pistoia}@us.ibm.com ABSTRACT We leverage combined concrete and symbolic execution and sev- eral fault-localization techniques to create a uniquely powerful tool for localizing faults in PHP applications. The tool automatically generates tests that expose failures, and then automatically local- izes the faults responsible for those failures, thus overcoming the limitation of previous fault-localization techniques that a test suite be available upfront. The fault-localization techniques we employ combine variations on the Tarantula algorithm with a technique based on maintaining a mapping between statements and the frag- ments of output they produce. We implemented these techniques in a tool called Apollo, and evaluated them by localizing 75 randomly selected faults that were exposed by automatically generated tests in four PHP applications. Our findings indicate that, using our best technique, 87.7% of the faults under consideration are localized to within 1% of all executed statements, which constitutes an almost five-fold improvement over the Tarantula algorithm. Categories and Subject Descriptors D.2.4 [Software Engineering]: Software/Program Verification ; D.2.5 [Software Engineering]: Testing and Debugging General Terms Reliability, Verification 1. INTRODUCTION Web applications are typically written in a combination of sev- eral programming languages, such as JavaScript on the client side, and PHP with embedded SQL commands on the server side. Such applications generate structured output in the form of dynamically generated HTML pages that may refer to additional scripts to be ex- ecuted. As with any program, programmers make mistakes and in- troduce faults, resulting in Web-application crashes and malformed dynamically generated HTML pages. While malformed HTML er- rors may seem trivial, and indeed many of them are at worst minor annoyances, they have on occasion been known to enable serious Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ICSE ’10, May 2-8 2010, Cape Town, South Africa Copyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00. attacks such as denial of service 1 . We present the first fully auto- matic tool that finds and localizes malformed HTML errors in Web applications that execute PHP code on the server side. In previous work [3, 4], we adapted the well-established tech- nique of combined concrete and symbolic execution [9, 25, 5, 10, 28] to Web applications written in PHP. With this approach, an ap- plication is first executed on an empty input, and a path condition is recorded that reflects the application’s control-flow predicates that have been executed that depend on input. Then, by changing one of the predicates in the path condition, and solving the resulting con- dition, new inputs can be obtained, and executing the program on these inputs will result in additional control-flow paths being exer- cised. In each execution, faults that are observed during the exe- cution are recorded. This process is repeated until either sufficient coverage of the statements in the application has been achieved, a sufficient number of faults has been detected, or the time bud- get is exhausted. Our previous work addresses a number of issues specific to the domain of PHP applications that generate HTML output. In particular: (i) it integrates an HTML validator to check for errors that manifest themselves by the generation of malformed HTML, (ii) it automatically simulates interactive user input, and (iii) it keeps track of the interactive session state that is shared be- tween multiple PHP scripts. We implemented these techniques in a tool called Apollo. In previous experiments on 6 open-source PHP applications, Apollo found a total of 673 faults [4]. A problem not addressed by our previous work is that it fails to pinpoint the specific Web-application instructions that cause exe- cution errors or the generation of malformed HTML code. Without that information, correcting these types of issues can be very diffi- cult. This paper addresses the problem of determining where in the source code changes need to be made in order to fix the detected failures. This task is commonly referred to as fault localization, and has been studied extensively in the literature [30, 15, 23, 16, 7, 14]. Our use of combined concrete and symbolic execution to obtain passing and failing runs overcomes the limitation of many existing fault-localization techniques that a test suite with passing and failing runs be available upfront. The fault-localization tech- niques explored in this paper combine variations on the Tarantula algorithm by Jones et al. [15, 14] with the use of an output map- ping. The first main ingredient of our combined approach is the Taran- tula algorithm by Jones, et al. [15, 14], which predicts statements that are likely to be responsible for failures. It does so by comput- ing, for each statement, the percentages of passing and failing tests that execute that statement. From this, a suspiciousness rating is computed for each executed statement. Programmers are encour- aged to examine the statements in order of decreasing suspicious- 1 For example http://support.microsoft.com/kb/810819.