Detecting Security Anomalies from Internet Traffic using the MA-RMSE Algorithms Breno Pinto¹, Varin Khera², Chun Che Fung³ 1. Computer Security Incident Response Team, BrasilTelecom, Brasília, DF, Brazil 2. Nokia Siemens Network, Australia 3. School of Information Technology, Murdoch University, Australia Email: { 1 breno.silva@gmail.com, 2 varin.khera@nsn.com, 3 l.fung@murdoch.edu.au} Abstract - Many detection techniques against worms, denial of service attacks and botnets on the Internet have been developed. It is difficult to detect these threats if the malicious traffic has insufficient intensity, which is usually the case. To make the problem worse, legitimate Internet services behaving like worm and complexity network environments undermines the efficiency of the detection techniques. This paper proposes an entropy- based Internet threats detection approach that determines and reports the traffic complexity parameters when changes in the traffic complexity content may indicate a malicious network event. Based on the experiment, the proposed method is efficient and produces less false positive and false negative alarms with a faster detection time. I. INTRODUCTION Today, a reliable information and communication system is essential to the smooth operation of most of the organizations. However, the world still witnesses recurring events such as virus, worms and cyber attackers for numerous reasons, despite the significant research and activities in the discipline of Information Security. This illustrates the inherent weaknesses in the current information and communication technologies and an urgent need to develop a better and more proactive approach to achieve heightened security for present day and next generation networks. In this paper, a proactive anomaly detection method based on network behavior is proposed. The proposal is called Measure of Anomaly – Representative Measure of String Entropy (MA-RMSE). In this method, the detection of network anomalies is achieved by analyzing the current network bits distribution level. In general, it could be assumed that should a change of the traffic pattern is observed, this will indicate a possible attack. MA-RMSE uses a flexible and fast approach to estimate traffic distribution, by computing the entropy and statistical properties of the network traffic. Given the known history of normal traffic, it is possible to distinguish anomalies that change the Internet traffic bits distribution abruptly or slowly thereby indicating the occurrence of anomalies, and possible malicious activities. Normally, entropy-based algorithms designed for Internet threats detection require a significant amount of malicious packets which cause rapid changes in the traffic pattern, before the detection algorithms are able to report the anomaly among the traffic [5]. This however is inefficient in detecting new forms of attacks, this paper therefore aims at presenting a new algorithm that provides a faster detection time and improved accuracy. This paper is organized as follows. Section II provides an introduction to the background of the entropy and detection algorithms. Section III discusses the implementation and related issues. Section IV presents some results based on some known attacks. Section V concludes the paper with discussion on further work. II. ENTROPY AND DETECTION ALGORITHMS The MA-RMSE technique is based on the measurement of the complexity of the Internet traffic. This is an application of the Information Entropy approach [1] and the resultant parameters will define the statistical properties of the traffic. These values in turn can be used to differentiate between normal and anomaly traffic patterns. In the following section, the general entropy theories of Shannon entropy, and the Kolmogorov complexity, which form the foundation of the proposed MA-RMSE algorithm, are introduced. A. Shannon Entropy theory Entropy is a measurement of the uncertainty associated with a random variable [1]. The term by itself in this context usually is referred to as Shannon Entropy. This value quantifies in the form of an expected value, the information contained in a message, usually in units such as bits. Entropy H(x) can be described as: H ( x) =− p( x) Logp( x) (1) Where Log is the logarithm in base 2, which determines the degree of chaotic distribution of probability p and x is a string of bits. Traditional entropy traffic detection mechanism looks for the distribution of source addresses, destination addresses, ports, flow and correlates them to detect an anomaly [2,3,4]. This method works well in detecting distributed type of attack such as denial of service attacks, port scans, and large worm propagations pattern. However, the method is insufficient in detecting modern attacks which do not change the traffic patterns abruptly. Also, detection becomes difficult when the attack uses more than one protocol types and changes between the ports, and incurred other variations [5]. 887 978-1-4244-3760-3/09/$25.00 c  2009 IEEE Authorized licensed use limited to: Murdoch University. Downloaded on October 15, 2009 at 03:23 from IEEE Xplore. Restrictions apply.