How Does Execution Information Help with Information-Retrieval Based Bug Localization? Tung Dao Computer Science Virginia Tech Blacksburg, VA 24060 tungdm@vt.edu Lingming Zhang Computer Science The University of Texas at Dallas Dallas, TX 75080 lingming.zhang@utdallas.edu Na Meng Computer Science Virginia Tech Blacksburg, VA 24060 nm8247@cs.vt.edu Abstract—Bug localization is challenging and time-consuming. Given a bug report, a developer may spend tremendous time comprehending the bug description together with code in order to locate bugs. To facilitate bug report comprehension, informa- tion retrieval (IR)-based bug localization techniques have been proposed to automatically search for and rank potential buggy code elements (i.e., classes or methods). However, these techniques do not leverage any dynamic execution information of buggy programs. In this paper, we perform the first systematic study on how dynamic execution information can help with static IR-based bug localization. More specifically, with the fixing patches and bug reports of 157 real bugs, we investigated the impact of various execution information (i.e. coverage, slicing, and spectrum) on three IR-based techniques: the baseline technique, BugLocator, and BLUiR. Our experiments demonstrate that both the coverage and slicing information of failed tests can effectively reduce the search space and improve IR-based techniques at both class and method levels. Using additional spectrum information can further improve bug localization at the method but not the class level. Some of our investigated ways of augmenting IR-based bug localization with execution information even outperform a state-of-the-art technique, which merges spectrum with an IR- based technique in a complicated way. Different from prior work, by investigating various easy-to-understand ways to combine execution information with IR-based techniques, this study shows for the first time that execution information can generally bring considerable improvement to IR-based bug localization. I. I NTRODUCTION Bug localization, or fault localization, is important in soft- ware maintenance, because effective bug fixing relies on precise bug location information. However, given a bug report and a buggy program, developers may spend tremendous time and effort understanding the bug description and code to locate bugs. To facilitate bug comprehension and accelerate bug finding, researchers have proposed various IR-based bug localization techniques [36], [21], [19], [20]. By treating the bug report as a query, and the source code files as plain docu- ments, these techniques rank software entities (i.e., classes or methods) based on their relevance or similarity to the query. The more relevant a program entity is, the higher it is ranked as a potential bug location. These IR-based techniques can facilitate bug localization and program comprehension, because they help developers focus effort on bug-relevant code elements. In particular, Zhou et al. proposed BugLocator to use a specialized Vector Space Model (VSM), called rVSM, by considering file lengths and bug history [36]. They demonstrated that rVSM outperformed other IR models on real bugs from four open-source projects. Saha et al. further proposed BLUiR [21] to use another revised VSM by considering code constructs, such as class and method names. Their experiments showed that BLUiR even outperformed BugLocator. Despite the various IR-based techniques, we are curious whether the execution information of buggy programs can further help bug localization and program comprehension. Le et al. proposed the first tool, AML, to combine IR-based bug localization with spectrum execution information [13]. They used a hybrid model to encode both spectrum and textual information into a specialized VSM. They found that AML outperforms Learning-to-rank [32] (a state-of-the-art IR-based bug localization technique), and MULTRIC [30] (a state-of- the-art spectrum-based bug localization technique). However, it is still unknown how various types of execution information can generally help with IR-based bug localization. To systematically investigate the impact of various execu- tion information on IR-based techniques, we performed an extensive study on three kinds of execution information, and three state-of-the-art IR-based techniques, using an existing dataset of 157 real bugs. More specifically, we investigated the following three types of information: (1) coverage—the classes or methods covered by failed tests, (2) slicing—the classes or methods in the dynamic slice [29] of each failure witness statement (i.e., a failure assertion or an exception- throwing statement), and (3) spectrum—the suspiciousness score of each executed class or method, which describes the coverage ratio between passed and failed tests [10]. Hypotheti- cally, coverage and slicing may help with IR-based techniques by refining the search space. The reason is if an entity (i.e., class or method) is not covered by a failed test or does not occur in the slice of a failure witness statement, it is unlikely to be buggy. Spectrum information may further help by ranking program entities purely based on suspiciousness scores. Its ranking can complement the ranking by IR-based techniques. In this study, we experimented with three existing IR-based techniques: the baseline, BugLocator [36], and BLUiR [21]. To assess the impact of different execution information on 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241 2017 IEEE 25th International Conference on Program Comprehension (ICPC) 978-1-5386-0535-6/17 $31.00 © 2017 IEEE DOI 10.1109/ICPC.2017.29 241