IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 5, NO. 4, DECEMBER 2010 613 Fuzzy Classification Metrics for Scanner Assessment and Vulnerability Reporting Peter Kok Keong Loh, Senior Member, IEEE, and Deepak Subramanian Abstract—In information security, web application scanners de- tect and provide some diagnoses for specific vulnerabilities. How- ever, scanner performance as well as the damage potential of dif- ferent vulnerabilities varies. This undermines the development of effective remediation solutions and the reliable sharing of vulner- ability information. This paper describes an approach based on soft computing technology for the development of metrics that are used to grade web application scanners and vulnerabilities so that scanner performance can be evaluated and confidence levels can be computed for vulnerability reports. These metrics help derive a level of assurance that will support security management decisions, enhance effective remediation efforts, and could serve as security tool design metrics. Index Terms—Confidence level, Fuzzy classifiers, scanner, vul- nerability, web application. I. INTRODUCTION T HE Internet has become one of the most significant forms of information storage, communication, and commerce leading to significant growth in web-based systems. Its popu- larity has also triggered the rise in occurrence of software vul- nerabilities that can disrupt several such systems [16]. In infor- mation security for web-based applications, the detection and diagnosis of software vulnerabilities are thus important tasks for the user. A standard technique is to use web application scanners to detect these vulnerabilities and suggest possible diagnoses. However, there are often significant differences in the content, organization, and format of different vulnerability reports [18]. Furthermore, the capabilities of existing scanners vary greatly depending on the design and complexity of the implemented scanner algorithms. These factors undermine the reliability of diagnostics produced by scanners thereby impeding effective re- mediation as well as the sharing and utilization of vulnerability reports. In the detection and diagnoses of different web application vulnerabilities, it must also be noted that some vulnerabilities are more dangerous than others in terms of potential damage/ risks [10]. This emphasizes the need for suitable metrics to grade the various vulnerabilities as well as the different scanners. The Manuscript received February 19, 2010; revised August 12, 2010; accepted August 23, 2010. Date of publication September 13, 2010; date of current ver- sion November 17, 2010. This work was supported by the Ministry of Defence (Singapore). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Darko Kirovski. The authors are with the Computer Security Laboratory, Nanyang Technolog- ical University, Singapore 639798, Singapore (e-mail: askkloh@ntu.edu.sg). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TIFS.2010.2075926 challenge then is to develop suitable metrics towards this end. Besides enhancing the assurance level of scanner diagnostics, such metrics enable more effective security management deci- sions and may lend themselves to act as design parameters in software security tool designs. Our approach described in this paper focuses on web application vulnerabilities and is based on fuzzy metrics that are determined empirically from the scan- ners used. A system framework has been proposed to achieve standard- ization of scanner outputs across different web technologies [2]–[4]. Together with a suitable set of rules and a data miner, standardized scanner reports can be evaluated to give required user reports. This framework also supports the realization of a vulnerability detection engine that is scalable to different scanners as well as web technology. This paper focuses on the research work done in designing fuzzy metrics that will support the grading of vulnerability and scanners for computation of confidence levels in the report generation subsystem of this framework. The rest of the paper is organized as follows. Section II gives the motivation behind this research while Section III presents a review of existing related research. Section IV describes the pre- liminaries while Section V details the design of the fuzzy clas- sification metrics. Section VI presents the design of the scanner and vulnerability grading systems. Section VII provides the de- tailed methodology to calculate the first and second degree con- fidence levels for vulnerability reports while Section VIII pro- vides an illustrative example to describe the working of the framework. Section IX concludes the paper followed by refer- ences and acknowledgements. II. RESEARCH MOTIVATION AND POTENTIAL IMPACT Existing web application scanners generate vulnerability di- agnoses and remediation that are typically targeted at known vulnerabilities and specific scenarios. The occurrence and be- havior of web vulnerabilities are, however, inherently uncertain. While data classification can act as an enabler for a more effec- tive diagnosis and calculation of daily vulnerability exposure (DVE) helps in determining patch effectiveness [12], the widely varying detection capabilities encountered in existing scanners as well as the differing threat/risk levels posed by individual vul- nerabilities have not been addressed. Hence, it becomes a chal- lenge to be able to utilize scanner-generated diagnostics with confidence. One solution is to develop a systematic approach to grade vulnerability impact and scanner performance. The results can then be used to provide a level of assurance that will sup- port the use and sharing of vulnerability information for security management decisions as well as enhance remediation efforts. 1556-6013/$26.00 © 2010 IEEE