On the symbiosis of specification-based and anomaly-based detection Natalia Stakhanova a, *, Samik Basu b , Johnny Wong b a Faculty of Computer Science, University of New Brunswick Fredericton, NB E3B 5A3, Canada b Department of Computer Science, Iowa State University Ames, IA 50011, USA article info Article history: Received 25 June 2009 Received in revised form 28 August 2009 Accepted 31 August 2009 Keywords: Specification-based approach Anomaly detection Program behavior specification Network monitoring Intrusion detection abstract As the number of attacks on computer systems increases and become more sophisticated, there is an obvious need for intrusion detection systems to be able to effectively recognize the known attacks and adapt to novel threats. The specification-based intrusion detection has been long considered as a promising solution that integrates the characteristics of ideal intrusion detection system: the accuracy of detection and ability to recognize novel attacks. However, one of the main challenges of applying this technique in practice is its dependence on the user guidance in developing the specification of normal system behavior. In this work, we present an approach for automatic generation of specifications for any software systems executing on a single host based on the combination of two techniques: specification-based and anomaly-based approaches. The proposed technique allows automatic development of the normal and abnormal behavioral specifications in a form of variable-length patterns classified via anomaly-based approach. Specifically, we use machine-learning algorithm to classify fixed-length patterns generated via sliding window technique to infer the classification of variable-length patterns from the aggre- gation of the machine learning based classification results. We describe the design and implementation of our technique and show its practical applicability in the domain of security monitoring through simulation and experiments. ª 2009 Elsevier Ltd. All rights reserved. 1. Introduction The rapid increase in the number, sophistication and impact of computer attacks makes the computer systems unpre- dictable and unreliable, emphasizing the importance of intrusion detection ability to correctly recognize known attacks and identify new threats. Typically, intrusion detection refers to a variety of tech- niques for detecting attacks in the form of malicious and unauthorized activities. There are three broad categories of detection approaches (Sekar et al., 2002) (a) misuse-based (b) anomaly-based and (c) specification-based. Misuse-based technique relies on pre-specified attack signatures, and any execution sequence matching with a signature is flagged as abnormal. An anomaly-based approach, on the other hand, depends on automatic classification of executions as normal patterns, and any deviation from normal patterns is classified as malicious or faulty. Unlike misuse-based detection, anomaly-based techniques can detect previously unknown abnormalities. However, anomaly-based approaches rely on statistical or machine learning classification techniques which can only classify (usually) pre-specified, fixed-length behavioral patterns, and suffer from the disadvantage of a high rate of false positives (Lazarevich et al., 2003). Specifi- cation-based techniques operate in a similar fashion to anomaly-based method detecting deviations from the * Corresponding author. E-mail address: nstakhanova@gmail.com (N. Stakhanova). available at www.sciencedirect.com journal homepage: www.elsevier.com/locate/cose ARTICLE IN PRESS 0167-4048/$ – see front matter ª 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.cose.2009.08.007 computers & security xxx (2009) 1–16 Please cite this article in press as: Stakhanova N et al., On the symbiosis of specification-based and anomaly-based detection, Comput. Secur. (2009), doi:10.1016/j.cose.2009.08.007