Application of anomaly detection algorithms for detecting SYN flooding attacks * Vasilios A. Siris * ,1 , Fotini Papagalou Institute of Computer Science, Foundation for Research and Technology-Hellas (FORTH), P.O. Box 1385, GR 711 10 Heraklion, Crete, Greece Received 21 January 2005; received in revised form 30 August 2005; accepted 16 September 2005 Available online 19 October 2005 Abstract We investigate statistical anomaly detection algorithms for detecting SYN flooding, which is the most common type of Denial of Service (DoS) attack. The two algorithms considered are an adaptive threshold algorithm and a particular application of the cumulative sum (CUSUM) algorithm for change point detection. The performance is investigated in terms of the detection probability, the false alarm ratio, and the detection delay, using workloads of real traffic traces. Particular emphasis is on investigating the tradeoffs among these metrics and how they are affected by the parameters of the algorithm and the characteristics of the attacks. Such an investigation can provide guidelines to effectively tune the parameters of the detection algorithm to achieve specific performance requirements in terms of the above metrics. q 2005 Elsevier B.V. All rights reserved. Keywords: Denial of service; Change point detection; Network security 1. Introduction Over the past few years many sites on the Internet have been the target of Denial of Service (DoS) attacks, among which TCP SYN flooding is the most prevalent [10]. Studies 2 show an increase of DoS attacks in the last few years, which can result in disruption of services that costs from several millions to billions of dollars. The aim of denial of service attacks are to consume a large amount of resources, thus preventing legitimate users from receiving service with some minimum performance. TCP SYN flooding exploits TCP’s three-way handshake procedure, and specifically its limitation in maintaining half-open connections. Any system connected to the Internet and providing TCP-based network services, such as a Web server, FTP server, or mail server, is potentially a target of such an attack. A TCP connection starts with the client sending a SYN message to the server, indicating the client’s intention to establish a TCP connection. The server replies with a SYN/ACK message to acknowledge that it has received the initial SYN message, and at the same time reserves an entry in its connection table and buffer space. After this exchange, the TCP connection is considered to be half open. To complete the TCP connection establishment, the client must reply to the server with an ACK message. In a TCP SYN flooding attack, an attacker, from a large number of compromised clients in the case of distributed DoS attacks, sends many SYN messages, with fictitious (spoofed) IP addresses, to a single server (victim). Although the server replies with SYN/ACK messages, these messages are never acknowledged by the client. As a result, many half- open connections exist on the server, consuming its resources. This continues until the server has consumed all its resources, hence can no longer accept new TCP connection requests. Recently, end-system approaches have been proposed for protection against SYN flooding attacks. However, such approaches require modifications to end-systems and cannot protect against attacks that proceed with full TCP handshaking. Moreover, there is still debate on the potential overhead that can be introduced by such end-system approaches. A common feature of DoS attacks is that they lead to changes in a measured statistic of a network traffic flow. Such statistics can include the type and size of packets, the number of half open connections, and the rate of packets associated with a particular application or port number; in the case of TCP SYN flooding the statistic is the number of TCP SYN packets. Computer Communications 29 (2006) 1433–1442 www.elsevier.com/locate/comcom 0140-3664/$ - see front matter q 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.comcom.2005.09.008 * This work was supported in part by the EC funded project SCAMPI (IST- 2001-32404). * Corresponding author. Tel.: C30 2810 391726; fax: C30 2810 391601. E-mail address: vsiris@ics.forth.gr (V.A. Siris). 1 The authors are also with the Dept. of Computer Science, University of Crete. 2 2002 and 2003 CSI/FBI Cybercrime Survey Report. The 2003 report indicates that DoS attacks alone were responsible for a loss of $65 million.