A Systematic Survey of CMM Experience and Results James D. Herbsleb Dennis R. Goldenson Software Engineering Institute, Carnegie Mellon University Pittsburgh, PA 15213 USA (412) 268-7389, (412) 268-8506 <jherbsle, dg>@ sei.cmu.edu Abstract The capability maturity model* (CMM) for sojlware has become very influential as a basis for software process improvement (SPI). Most of the evidence to date showing the results of these efforts has consisted of case studies. We present a systematic survey of organizations that have undertaken CMM-based SPI to get more representative results. We found evidence that process maturity is in fact associated with better organizational pe~ormance, and that software process appraisals are viewed, in retrospect, as extremely valuable and accurate guides for the improvement effort. The path was not always smooth, however, and e~orts generally took longer and cost more than expected. A number of factors that distinguished highly successful from unsuccessful efforts are identified. Most of these factors are under management control, suggesting that a number of specijic management decisions are likely to have a major impact on the success of the effort. Keywords: capability maturity model, software process improvement, software process appraisals 1: Introduction The Capability Maturity Model* (CMM) for software [1, 2] plays a major role as a reference model for software process improvement (SPI) efforts in hundreds of software organizations worldwide. The model is used as a standard for appraising the current state of the organization’s software process, as well as a guide for identifying and prioritizing the actions comprising the SPI effort. * Capability Maturity Model and CMM are service marks of Carnegie Mellon University. 1.1: The CMM The CMM describes five levels of process maturity. Each level has a number of key process areas (KPAs) that represent the primary issues the organization needs to address in order to mature its process. At the initial level of maturity, software projects rely on the skills, and too often the heroic efforts, of individual engineers. The risk of “runaway” projects is high, and developers are forced to lurch from one emergency to another. The KPAs for the repeatable level focus on effective project management, which aims to improve the ability to make and meet reasonable schedule and budget commitments. At the defined level, improvement focuses on developing tailorable software processes and other organizational assets, so that organizational learning can span projects. Next is the managed level, in which priority is given to monitoring software processes. Finally, in the optimizing level, quantitative data are consistently used for continuous process improvement. 1.2: Evaluating the CMM The CMM is not without its critics (e.g., [3]). It is sometimes claimed, for example, that adopting the CMM encourages too much bureaucracy, or that the CMM is incomplete or flawed. This debate is partly concerned with scope, policy issues, and ccmceptual questions, e.g., whether the model harmonizes appropriately with international standards such as 1S0-9000. But the debate also focuses on the supposed consequences of adopting the CMM as the basis for SPI efforts. Will the organization get bogged down in red tape or suffer other damage, or will it benefit and show improved performance? Unlike some other improvement models, tlhe CMM does not explicitly require the results of each change to be measured. It is here that valid ancl reliable empirical evidence is desperately needed. 0270-5257/96 $5.00 @ 1996 IEEE 323 Proceedings of ICSE-18