Histogram Based Payload Processing for Unsupervised Anomaly Detection Systems in Network Intrusion I˜ nigo Perona, I˜ naki Albisua, Olatz Arbelaitz, Ibai Gurrutxaga, Jos´ e I. Mart´ ın, Javier Muguerza, and Jes´ us M. P´ erez Dept. of Computer architecture and Technology University of the Basque Country M. Lardizabal, 1, 20018 Donostia, Spain {inigo.perona,inaki.albisua,olatz.arbelaitz,i.gurrutxaga,j.martin,j. muguerza,txus.perez}@ehu.es Abstract. The popularity of computer networks broadens the scope for network attackers and increases the damage these attacks can cause. In this context, any complete security package includes a network Intru- sion Detection System (nIDS). This work focuses on nIDSs which work by scanning the network traffic. We present a service-independent pay- load processing approach, based on histogram representation, to increase detection rates in non-flood attacks. We implemented three different op- tions combining histogram representation and fixed width clustering al- gorithm for anomaly detection, and compared them to a sysstem based on packets’ header information, another system based on ad hoc payload processing and our previous general payload processing proposal. The new options outperformed the previous ones; they detected efficiently most of the attack types. Moreover, the proper integration of the knowl- edge of the different techniques, payload-based and packet header-based, always improved the results. This work leads us to conclude that payload analysis can be used in a general manner, with no service- or port-specific modelling, to detect attacks in network traffic. Key words: Intrusion detection systems, unsupervised anomaly detec- tion, payload, histogram, AUC 1 Introduction There has been a huge increase in the use of computer networks. This fact broadens the scope for network attackers and increases the damage these at- tacks can cause. Network attacks affect the security of the information stored on computers connected to the network and its stability. Therefore, it is very important to build systems that are able to detect attacks before they cause damage. Any complete security package includes a network Intrusion Detection System (nIDS). The detection of network attacks can be done by human analysis or auto- matically. The detection by human analysis requires memorisation, looking up