Session F1G Work in progress - Measuring the ROI time for Static Analysis Walter W. Schilling, Jr. 1 and Dr. Mansoor Alam 2 Abstract— Static analysis is one method that offers potential for reducing errors in delivered software. Static analysis is currently used to discover buffer overflows and mathematical errors, as well as verifying compliance with documented programming standards. Static analysis is routinely used in safety critical soft- ware applications within the avionics and automotive industries. Outside of these applications, static analysis is not a routinely taught method for software development. This paper intends to provide a quantitative measure for evaluating the effectiveness of static analysis as well as presenting results from an academic environment. Index Terms— Computer science education, Programming en- vironments, Software economics, Software engineering, Software metrics, Software tools, Static Analysis I. I NTRODUCTION Static analysis represents one technique that has been pro- posed to aid in improving the quality of developed software. Static analysis, in this case, involves analyzing the source code developed for potential latent faults which may result in a future failure. This paper presents an approach for quantitatively measur- ing the benefit for static analysis on C source code, referred to as ROI time . Results are presented for a project which created a simulator for the OSI network model. The simulator was developed using the ISO C programming language, and the source code was intended to be compliant with the Misra C [1] coding standard. The project was used the PSP methodology and data was tracked with the Software Process Dashboard Project software. II. LITERATURE SURVEY Static analysis is routinely used in mission critical source code development. Giessen provides an overview of the con- cept of static analysis, including the philosophy and practical issues related to static analysis [2]. Robert Glass reports that static analysis can remove upward of 91% of errors within source code [3]. It has also been found effective at detecting pieces of dead or unused source code in embedded systems [4]. Bratt and Klemm indicate that static analysis can be ef- fective at finding defects, though there is often a scalability problem with large projects [5]. Venet and Brat discuss an 1 Walter W. Schilling, Jr. Electrical Engineering and Computer Science Department, The University of Toledo, Toledo, Ohio 43615, Email: wal- ter.schilling@computer.org 2 Dr. Mansoor Alam, Electrical Engineering and Computer Science Department, The University of Toledo, Toledo, Ohio 43615, mail: malam@eecs.utoledo.edu Planning Design Design Review Code Code Review Test Postmortum Analysis Static Analyze Source Code (New Phase) Compile Fig. 1. Process flow including static analysis. improved software tool for the analysis of embedded C appli- cations which does not appear to suffer from scalability issues and also has an improved noise factor in the static analysis [6]. Larochelle and Evans document a methodology that can be used to statically detect buffer vulnerabilities in code written in the C programming language [7]. A similar usage of static analysis is documented by Ganapathy. [8] III. PROJECT PROCESS OVERVIEW Development of the network simulator project started with a PSP Level 3 process. In order to avoid some of the pitfalls [9] of using PSP in an academic environment, as well as reduce the amount of paperwork generated, the project was managed using the Software Process Dashboard Project. The inclusion of a static analysis phase within the develop- ment process represents the one change from a standard PSP Level 3 project. For this project, consideration was given to the appropriate time to perform static analysis. Static analysis is typically used during the code review phase. However, for this project, it was decided that the appropriate location for static analysis would be following the compilation phase. At the completion of compilation, the code compiles without warning, and thus is free from syntax errors. By analyzing at this phase, a true estimate of the number of faults that would not have been detected by other means excepting testing can be obtained. IV. PRELIMINARY RESULTS Static analysis was found to be quite effective at removing programming errors from source code which had escaped detection at the review and compilation phase. Table I shows resulting defect injection, defect detection, effectiveness, and return on investment for each phase. The number of defects injected is a measure of the number of defects that can be traced back to being injected in a given phase. The number of defects detected is a count of the number of defects detected 0-7803-9077-6/05/$20.00 c 2005 IEEE 35th ASEE/IEEE Frontiers in Education Conference F1G-30 October 19 - 22, 2005, Indianapolis, IN