602 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 55, NO. 4, NOVEMBER 2008 A Contextual Approach to Improving Software Metrics Practices Helle Damborg Frederiksen and Lars Mathiassen, Member, IEEE Abstract—Even though empirical studies report less than satis- factory results with software metrics, we have limited knowledge on how to improve programs that are already implemented. On this backdrop, we report from a three-year collaborative action research project in Software, Inc., aimed at increasing the benefits of their metrics program. The perceived challenge was not to im- prove individual metrics, but rather to more effectively integrate the metrics program as a whole into the managerial and organi- zational context at Software, Inc. To organize the results, we used Pettigrew’s framework for studying organizational change, em- phasizing the content, the context, and the process of the improve- ment effort. We considered the content of improvement through an information-centric analysis of the program as a medium for inter- actions between stakeholders; this analysis focused on the usage of metrics and helped redesign the program. The context was taken into account through an organization-centric analysis of manage- ment practices, organizational structures, people, and technical systems involved in executing the metrics program; this analysis focused on operation of the program and helped understand how to implement the new program. Finally, the improvement process was guided by the IDEAL model for innovating software practices; this approach helped combine different analyses into a coherent change effort. The paper presents this contextual approach to improving software metrics practices; it presents the results of applying it in Software, Inc.; and offers lessons for how managers can use contextual approaches to increase the benefits of software metrics programs. Index Terms—Contextual approach, information-centric analy- sis, organization-centric analysis, software metrics improvement. I. INTRODUCTION S OFTWARE engineering has matured as a discipline [58] and the idea of measuring processes and outcomes has come to play an important role. One prominent example is Al- brecht’s invention of function points as a measure of software size [33]. [54] sets the basic terminology: “A measure provides a quantitative indication of the extent, amount, dimensions, ca- pacity, or size of some attribute of a product or a process. Measurement is the act of determining the measures.” Mea- sures result from collecting data and aggregating them through metrics. Software managers use the resulting performance in- Manuscript received March 14, 2006; revised June 2007, November 2007, and April 30, 2008. Current version published October 22, 2008. Review of this manuscript was arranged by Department Editor R. Sabherwal. This work was supported in part by the Department of Computer Science, Aalborg University, Denmark, and in part by The Case Company. H. D. Frederiksen is with the Department of Computer Science, Aalborg University, DK-9100 Aalborg, Denmark, and also with Software, Inc. (e-mail: hdf@cs.aau.dk). L. Mathiassen is with the Center for Process Innovation, J. Mack Robinson College of Business, Georgia State University, Atlanta, GA 30302-5029 USA (e-mail: lmathiassen@gsu.edu). Digital Object Identifier 10.1109/TEM.2008.2005547 dicators to support decision making and improvement initia- tives [8], [9], [25]. Much research has been dedicated to designing software met- rics [13], [20], [44], [54]. Another line of research has been con- cerned with implementation of metrics programs, e.g., [8], [27], and [30]. While many companies have designed and imple- mented metrics programs [16], [18], [20], [25], [27], [31], [52], most scholars agree that collecting data about software prac- tices and making these useful for managers is by no means trivial [16], [18], [24], [31], [52], [55]. In fact, recent studies suggest that the benefits from metrics programs are not as great as expected [18], and there are unexpected difficulties in col- lecting even a small number of straightforward metrics [28]. An estimated 78% of software metrics programs face serious difficulties or fail within the first two years [18]. It is therefore important to understand how existing metrics practices can be improved [46]. However, while a variety of approaches have been developed to assess software metrics programs, e.g., [8], [9], [21], [37], [44], and [46], we still have limited knowledge on how software organizations can improve their metrics practices. The purpose of this research is to explore how software or- ganizations can create a better fit between expected and ex- perienced benefits of their metrics programs. To that end, we investigated the improvement of a software metrics program based on action research principles [4], [38], [40], [45], [57]. This allowed us to combine practical problem solving in a soft- ware organization with scientific inquiry into how managers can improve existing software metrics programs. The project was carried out during 2000–2003 in Software, Inc., a large Danish software organization providing information technology (IT) services for municipal institutions. The project was carried out as a collaborative effort [32], [40] between the first author who is responsible for metrics at Software, Inc., the second author who is a software researcher, and colleagues at Software, Inc., involved in the initiative. The perceived challenge was not to rethink and redesign individual metrics, but rather to integrate the program as a whole more effectively into the managerial and organizational context at Software, Inc. As a consequence, we organized the results using Pettigrew’s framework for studying organiza- tional change [50], [51], emphasizing the content, the context, and the process of improving software metrics at Software, Inc. Pettigrew’s framework supports longitudinal investiga- tions of how transformation efforts unfold in organizational settings; outcomes are understood as resulting from interac- tions between the content, the context, and the process of the transformation under study; and, insights and lessons that link transformational patterns to outcomes are identified and 0018-9391/$25.00 © 2008 IEEE