A comparison of the efficiency and effectiveness of vulnerability discovery techniques Andrew Austin, Casper Holmgreen, Laurie Williams ⇑ Department of Computer Science, North Carolina State University, Raleigh 27695, USA article info Article history: Available online 8 December 2012 Keywords: Security Vulnerability Static analysis Penetration testing Black box testing White box testing abstract Context: Security vulnerabilities discovered later in the development cycle are more expensive to fix than those discovered early. Therefore, software developers should strive to discover vulnerabilities as early as possible. Unfortunately, the large size of code bases and lack of developer expertise can make discovering software vulnerabilities difficult. A number of vulnerability discovery techniques are available, each with their own strengths. Objective: The objective of this research is to aid in the selection of vulnerability discovery techniques by comparing the vulnerabilities detected by each and comparing their efficiencies. Method: We conducted three case studies using three electronic health record systems to compare four vulnerability discovery techniques: exploratory manual penetration testing, systematic manual penetra- tion testing, automated penetration testing, and automated static analysis. Results: In our case study, we found empirical evidence that no single technique discovered every type of vulnerability. We discovered that the specific set of vulnerabilities identified by one tool was largely orthogonal to that of other tools. Systematic manual penetration testing found the most design flaws, while automated static analysis found the most implementation bugs. The most efficient discovery tech- nique in terms of vulnerabilities discovered per hour was automated penetration testing. Conclusion: The results show that employing a single technique for vulnerability discovery is insufficient for finding all types of vulnerabilities. Each technique identified only a subset of the vulnerabilities, which, for the most part were independent of each other. Our results suggest that in order to discover the greatest variety of vulnerability types, at least systematic manual penetration testing and automated static analysis should be performed. Ó 2012 Elsevier B.V. All rights reserved. 1. Introduction Results of decades of empirical research on effectiveness and efficiency of fault and failure discovery techniques, such as unit testing and inspections, can be used to provide evidence-based guidance on the use of these techniques. However, similar empir- ical results on the effectiveness and efficiency of vulnerability dis- covery techniques, such as security-focused automated static analysis and penetration testing are sparse. As a result, practitio- ners lack evidence-based guidance on the use of vulnerability dis- covery techniques. In his book Software Security: Building Security In, Gary McGraw draws on his experience as a security researcher and claims: ‘‘Secu- rity problems evolve, grow, and mutate, just like species on a con- tinent. No one technique or set of rules will ever perfectly detect all security vulnerabilities’’ [1]. Instead, he advocates using a variety of vulnerability discovery and prevention techniques throughout the software development lifecycle. McGraw’s claim, however, is based upon his experience and is not substantiated with empirical evidence. The objective of this research is to aid in the selection of vul- nerability discovery techniques by comparing the vulnerabilities de- tected using each and comparing their efficiencies. In previous work [2], the first author analyzed four vulnerability discovery techniques on two electronic health record (EHR) sys- tems. The vulnerability discovery techniques analyzed included: exploratory manual penetration testing, systematic manual pene- tration testing, automated penetration testing, and automated sta- tic analysis. The first author used these four techniques on Tolven Electronic Clinician Health Record (eCHR) 1 and OpenEMR. 2 These two systems are currently used within the United States to store pa- tient records. Tolven eCHR and OpenEMR are web-based systems. This paper adds the same analysis conducted on an additional 0950-5849/$ - see front matter Ó 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.infsof.2012.11.007 ⇑ Corresponding author. E-mail addresses: andrew_austin@ncsu.edu (A. Austin), cmholmgr@ncsu.edu (C. Holmgreen), williams@csc.ncsu.edu (L. Williams). 1 http://sourceforge.net/projects/tolven/. 2 http://www.oemr.org/. Information and Software Technology 55 (2013) 1279–1288 Contents lists available at SciVerse ScienceDirect Information and Software Technology journal homepage: www.elsevier.com/locate/infsof