Detection of Network Buffer Overflow Attacks: A Case Study Barabas Maroš, Homoliak Ivan, Kačic Matej, Hanáček Petr Brno University of Technology, Faculty of Information Technology Božetěchova 2, CZ-612 66, Brno, Czech Republic {ibarabas, ihomoliak, ikacic, hanacek}@fit.vutbr.cz Abstract— This paper presents an automated detection method based on classification of network traffic using predefined set of network metrics. We proposed the set of metrics with focus on behavior of buffer overflow attacks and their sufficient description without the need of deep packet inspection. In this paper we describe two laboratory experiments of automated detection of buffer overflow attacks on vulnerable network services and their description by proposed set of network metrics. We present the principles of several chosen network metrics and their application on experimental attacks according to their nature in comparison to valid communication. Keywords— buffer overflow, detection, ids, network metrics I. INTRODUCTION The aim of our work is to design and develop a novel real- time intrusion prevention system for detection of advanced network attacks using behavioral characteristics of network communication. In previous articles [1][2] we proposed an idea of framework architecture that would be used for detection of various network threats. The papers presented the novel Automated Intrusion Prevention System (AIPS) which uses honeypot systems for the detection of new attacks and the automatic generation of behavioral signatures based on network flow metrics. The detection method is based on extraction of partial communication and creation of behavioral signature from the network flow by previously defined set of network metrics. We have successfully experimented with the architecture of the AIPS system and we defined 167 metrics divided into five categories according to their nature. These metrics are used to describe properties of detected attack not upon the fingerprint of common signature, but based on its behavior. Metrics are formally specified and extraction of them can be generally realized for each data flow. The specification includes statistic, dynamic, localization and especially behavioral properties of network communication. For the learning phase of classification, we use simulated and captured set of buffer overflow attacks and new attacks extracted from shadow honeypots deployed in protected network. These shadow honeypots are capable of detection either previously unknown buffer overflows using dynamic taint analysis [4]. This paper discusses two use cases of buffer overflow attacks simulated in laboratory conditions and their behavioral characteristics created by previously published set of network metrics. Together with the attack communications we present valid communications for further comparison of detection approaches and to emphasize the particular metrics used for the attack detection. We compare various chosen network metrics to visualize methods of detection buffer overflow attack vectors and their differences from valid communication. We chose several metrics that approximate the communication and visualized them (e.g. polynomial approximation of output communication in output direction from the side of attacked machine). The second contribution of this paper is the possibility of detection new zero-day attacks by successful and sufficiently abstract description of these attacks to detect the similarities in behavioral characteristics of various attack vectors of the same type (e.g. buffer overflow). II. METHOD OF DETECTION For classification of malicious attack vectors in network flow, the captured traffic must proceed through several steps of extraction as follows: 1) First, the captured traffic from mirrored network interface is divided into separate connections by meeting specific criteria (e.g. three-way handshake). Metrics extraction process considers communications data stored in libpcap format [3]. This process is closely defined in our previous article [1]. 2) The second step is the extraction of metric data from from each TCP connection by predefined set of network metrics. The output of the extraction is an array of integers, each number representing one metric or its part (for more complex metrics as polynomial representation, etc.) 3) All these metrics (extracted data) create the behavior signature of the particular connection that can be matched with known attack patterns. The comparison with the database is done by using artificial intelligence to suppress threshold evasion methods used by attackers. 4) For each matched attack the strongest features used for the detection are strengthen inside the detection model to increase the detection ratio (e.g. by increasing its weight). The initial set of attack signatures is gathered and later updated by shadow honeypot systems deployed within the monitored network. Honeypot systems are very good source of attack vectors due to their nature of detecting buffer overflow issues within the monitored services. We use honeypot system