Received: 13 July 2018 Revised: 14 August 2018 Accepted: 26 August 2018 DOI: 10.1002/cpe.4999 SPECIAL ISSUE PAPER Enhanced transductive support vector machine classification with grey wolf optimizer cuckoo search optimization for intrusion detection system E.M. Roopa Devi 1 R.C. Suganthe 2 1 Department of Information Technology, Kongu Engineering College, Erode, India 2 Department of Computer Science & Engineering, Kongu Engineering College, Erode, India Correspondence E.M. Roopa Devi, Assistant Professor Department of Information Technology Kongu Engineering College, Erode, India. Email: roopasen5@gmail.com Summary These days, the Intrusion detection System (IDS) is the most talked topic among the scientist and researchers and many research is going on in IDS, which is firmly connected to the protected utilization of system administrations. IDS are an essential part of the security infrastructure. The previous research works are focused to detect the attacks efficiently but it is failed to pro- duce more accurate classification results. To stay away from the previously mentioned issues, in the proposed framework, Hybrid Grey Wolf optimizer Cuckoo Search Optimization (HGWCSO) along with Enhanced Transductive Support Vector Machine (ETSVM) is proposed. This exploration incorporates the modules are, for example, preprocessing, selection of features and classification of features. The processing of data is done by using normalization technique by using min-max technique the main work is to replace the value missed and filters the features from NSL KDD dataset values. The main objective of processing of data is to increase the accuracy of classifica- tion. Then, the more relevant and optimal features are selected by using HGWCSO. The GWO robustness and searching performance is increased by cuckoo search algorithm. Then, the classifi- cation is performed to identify the intrusion attack types using ETSVM algorithm more efficiently. This classification algorithm is used to improve the attack detection accuracy higher. The exper- imental result concludes that the proposed HGWCSO with ETSVM algorithm provides better performance metrics in terms of high precision, sensitivity, specificity, and accuracy than the previous algorithms. KEYWORDS classification, data mining, ETSVM, feature selection, HGWCSO, intrusion detection 1 INTRODUCTION Intrusion detection has constantly assumed a critical part in computer security research. Two general ways to deal with to intrusion detection are such as misuse and anomaly detection. In misuse detection, essentially an example coordinating strategy, a client's exercises are contrasted and the known mark example of meddlesome assaults. In anomaly detection, looks for patterns that depart from the normal attributes. Disregarding their ability of identifying unknown attacks, anomaly detection system suffer from the essential trouble in characterizing what is normal properties. 1 Host based Intrusion Detection System (HIDS) is characterized as the way toward checking, gathering, and investigating occasions in a host to rec- ognize interruptions, which are infringement of data frameworks security strategies, prompting ruptures in confidentiality, integrity or availability. HIDS should thus be ready to determine such behavior. Misuse detective work approach will faithfully establish intrusion attacks within the cause of well-known signatures of discovered vulnerabil- ities. However, the protection department consultants raised interposition to outline rules or signature that limits the applying misuse detection to create intelligent IDS. On the opposite hand, the bizarre person police investigation advance deals with the applied mathematics analysis and pattern recognition issues. It is declared that this can be ready to notice novel attack with none previous knowledge for the classification model which will extract invasion pattern and data throughout preparation. 2 Host based Intrusion Detection System (HIDS) is characterized as the way Concurrency Computat Pract Exper. 2018;e4999. wileyonlinelibrary.com/journal/cpe © 2018 John Wiley & Sons, Ltd. 1 of 11 https://doi.org/10.1002/cpe.4999