Predicting Failures with Developer Networks and Social Network Analysis Andrew Meneely 1 , Laurie Williams 1 , Will Snipes 2 , Jason Osborne 3 1 Department of Computer Science, North Carolina State University, Raleigh, NC, USA {apmeneel, lawilli3}@ncsu.edu 2 Nortel Networks, Research Triangle Park, NC, USA. wbsnipes@nortel.com 3 Department of Statistics, North Carolina State University, Raleigh, NC, USA jaosborn@ncsu.edu ABSTRACT Software fails and fixing it is expensive. Research in failure prediction has been highly successful at modeling software failures. Few models, however, consider the key cause of failures in software: people. Understanding the structure of developer collaboration could explain a lot about the reliability of the final product. We examine this collaboration structure with the developer network derived from code churn information that can predict failures at the file level. We conducted a case study involving a mature Nortel networking product of over three million lines of code. Failure prediction models were developed using test and post-release failure data from two releases, then validated against a subsequent release. One model’s prioritization revealed 58% of the failures in 20% of the files compared with the optimal prioritization that would have found 61% in 20% of the files, indicating that a significant correlation exists between file- based developer network metrics and failures. Categories and Subject Descriptors D.2.8 [Software Engineering]: Metrics – process metrics, product metrics. General Terms Reliability, Human Factors, Verification Keywords Social network analysis, negative binomial regression, logistic regression, failure prediction, developer network 1. INTRODUCTION Software fails and fixing it is expensive. If testers can find software failures early in the software development lifecycle, the estimated cost of fixing the software dramatically decreases [10]. Research in failure prediction has provided many models to assess the failure-proneness of files, and have been highly successful at predicting software failures [3, 8, 11, 21, 22, 24, 25, 28]. Few models, however, consider the key cause of failures in software: people. People develop software and people test software. For large software systems, many people need to work together to develop software. This collaboration has a structure – a structure governed by elements of human social interaction and software development processes. Understanding the structure of developer collaboration could tell us a lot about the reliability of the final product. We examine this collaboration structure using a software development artifact common to most large projects: code churn information taken from revision control repositories. Code churn information has provided valuable metrics for failure prediction [21]. For example, a file with many recent changes tends to be more failure-prone than an unchanged file. But what if that file was updated by a developer who has worked with a lot of other developers? Maybe a “well-known” developer is less failure-prone. Code churn information can also tell us how these developers collaborated: we know who worked on what and when. From there, we can form a social network of developers (also known as a developer network) who have collaborated on the same files during the same period of time. Social Network Analysis (SNA) quantifies our notion of “well-known” developers with a class of metrics known as “centrality” metrics. The advantage of this developer network is that it provides a useful abstraction of the code churn information. With careful interpretation, one can use a developer network mid-development to identify potential risks and to guide verification and validation (V&V) activities such as code inspections. Our research goal is to examine human factors in failure prediction by applying social network analysis to code churn information. Failure prediction models have been successful for other areas (such as static analysis [16]), so the empirical techniques of model selection and validation have all been used with static code metrics [20]. We introduce file-based metrics based on SNA as additional predictors of software failures. A case study was conducted of a large Nortel networking product consisting of over 11,000 files and three million lines of code to build and evaluate the predictive power of network metrics. System test and post-release failure data from Nortel’s source repositories and defect tracking system were used in our study. The rest of this paper is organized as follows: Section 2 summarizes the background of Social Network Analysis and related work in failure prediction and developer networks. Section 3 introduces our developer networks, their associated metrics, and the analysis in failure prediction. Sections 4 and 5 summarize our case study of the Nortel product. Sections 6 and 7 summarize our work and outlines future work, respectively. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGSOFT 2008/FSE-16, November 9--15, Atlanta, Georgia, USA Copyright 2008 ACM 978-1-559593-995-1 ...$5.00.