Application Attack Detection System (AADS): An Anomaly Based Behavior Analysis Approach Ram Prasad Viswanathan, Youssif Al-Nashif, Salim Hariri NSF Center for Autonomic Computing Department of ECE, University of Arizona, Tucson Tucson, AZ, U.S.A {rampv, alnashif, hariri}@ece.arizona.edu Abstract—Network security, especially application layer security has gained importance with the rapid growth of web-based applications. Anomaly based approaches that profile the network traffic and look for abnormalities are effective against zero-day attacks. The complex nature of the web traffic, availability of multiple applications, privacy concerns and its own limitations make the development of such anomaly-based systems difficult. This paper proposes a framework for application layer anomaly detection. The framework uses a multiple model approach to detect anomalies. The framework encompasses a dedicated training phase to model the specific network traffic and a detection phase that can be deployed in real time. The framework has been applied to HTTP application traffic and multiple models have been developed. The experimental evaluation results of the AADS using multiple attack vectors have achieved a detection rate of almost 100%. In addition, the AADS has a false positive rate of 0.03%. Keywords: HTTP, anomaly, framework, multiple models, segregation I. INTRODUCTION The exponential growth of the World Wide Web (1 trillion Uri‟s indexed by Google as of 2008[1]) has led to an increase in network-based systems. With the advent of technologies such as Cloud Computing and Web services, traditional desktop applications have been moved to the Internet. This rapid growth of internet-based systems has also led to a corresponding increase in vulnerabilities and attacks. As noted by MITRE, vulnerabilities related to web applications have risen sharply [2] with XSS and buffer overflows forming majority of the attack patterns. With an increase in financial transactions and a corresponding increase in user-specific data online (social networking), the monetary aspect plays a significant role in web attacks. This has been augmented by the increase in software vulnerabilities that can be exploited by any attacker. Such a scenario has necessitated the development of network monitoring systems that can detect attacks. These systems, commonly known as intrusion detection systems (IDS) can be broadly classified into signature based and anomaly based systems. Signature based systems such as SNORT [3] look at patterns in the network payloads to detect attacks. These systems provide detection when the pattern is known beforehand. But signature-based systems are limited by their capabilities to detect novel attacks or those that do not have a fixed pattern. Anomaly based systems on the other hand look at profiling the normal traffic. Any deviation from normal behavior is considered an anomaly and an alert is generated. These systems are limited by the complexity of modeling the network activity, the running times of the detection routines and the high rate of false positives [4]. In our application anomaly analysis framework, models can be developed to detect various forms of attacks such as buffer overflows, XSS etc. Such a framework will also allow the system to model the various types of application traffic separately. Within each type, the traffic is classified into various objects. We have implemented anomaly detection models for the headers and text/javascript objects present in HTTP traffic. These models can detect a majority of web attacks including overflows and XSS. Due to space limitation, the Javascript object implementation will not be discussed in this paper. Each object has multiple models to characterize the normal behavior. This approach allows the flexibility to model each aspect of the object separately and at the same time give a weighted decision. The system employs dedicated training and detection phases. Our approach integrates the desirable aspects of both categories of intrusion detection systems. Similar to signature- based systems [5], the time complexity of our payload behavior analysis is linear with the input data size. Furthermore, our approach allows us to detect zero day attacks. The rest of the paper is organized as follows. Section 2 describes the related work carried out in application layer anomaly detection. Section 3 presents an overview of the framework and presents insights into the development of objects and modeling methodology. Section 4 describes the models developed for HTTP headers while section 5 deals with the evaluation of the system. Section 6 presents the conclusion and outlines the future work. The research presented in this paper is supported in part by National Science Foundation via grants numbers IIP-0758579, CNS- 0855087, and IIP-1032048, and it‟s conducted as part of the NSF center for Autonomic Computing at University of Arizona