Vol.:(0123456789) Wireless Personal Communications (2021) 121:659–686 https://doi.org/10.1007/s11277-021-08655-1 1 3 Intrusion Detection System Based on Hybrid Hierarchical Classifers Noor Mohd 1,2  · Annapurna Singh 1  · H. S. Bhadauria 1 Accepted: 14 June 2021 / Published online: 21 June 2021 © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021 Abstract According to this research work, the updated KDD-99 database is considered for the enlargement of hybrid hierarchical intrusion detection system (IDS). A total set of 4,898,431 testing instances comprising of 972,781 testing instances of normal type class and 3,925,650 testing instances of attack type class are used. The attack class consists of four distinct type of malicious activities named as DOS, U2R, R2L, and probing. The complete set of instances are further bifurcated into training and testing instance set in the ratio of 50–50. In hierarchical classifer structure, level-1 classifer is used for classifcation between normal and attack class. Attack class test samples are passed to level-2 classifer, which is used to identify the input test samples into DoS and additional type class. After that, other type test samples are passed to level-3 classifer, which is capable of classifying the tests into R2L and remaining class. Once again remaining class test samples are passed to level-4 classifer, which has the ability to classify the test samples into U2R and prob- ing type of attack. Then, the most excellent performing classifers at one and all level are again arranged in required hierarchical order to get hybrid hierarchical classifer, so that overall detection ratio is high at each level. After the validation of the proposed work on KDD-99 dataset, the highest detection rate is achieved with the help of hierarchical struc- ture of SSVM classifer based IDS i.e. 97.91%. It has also been calculated that the Overall Detection Accuracy (ODA) of 96.80%, 96.32%, 95.86%, 97.89% and 97.74% is achieved by SVM, PNN, DT, NFC and kNN classifers in hierarchical structure respectively. The proposed hybrid hierarchical classifer based IDS attained the ODA of 98.79%, which is highest among all experiments ODAs. Keywords Intrusion detection system · Hierarchical classifcation system · Support vector machine · Decision tree · Smooth support vector machine (SSVM) · k-Nearest neighbor classifers · Neuro fuzzy classifer · Probabilistic neural network * Noor Mohd noormohdcs@gmail.com; decentnoor@redifmail.com Annapurna Singh annapurnasingh78@gmail.com H. S. Bhadauria hsb76iitr@gmail.com 1 G. B. Pant Institute of Engineering and Technology, Pauri Garhwal, Uttarakhand, India 2 Graphic Era Deemed To Be University, Dehradun, Uttarakhand, India