249 International Journal of Communication Networks and Information Security (IJCNIS) Vol. 13, No. 2, August 2021 Novel Approach for IP-PBX Denial of Service Intrusion Detection Using Support Vector Machine Algorithm Abdirisaq M. Jama 1 , Othman O. Khalifa 2 and Nantha Kumar Subramaniam 3 1 Cluster of Applied Sciences, Open University Malaysia. Kuala lumpur, Malaysia 2 Department of Electrical and Computer Engineering, International Islamic University Malaysia 3 Cluster of Applied Sciences, Open University Malaysia. Kuala lumpur, Malaysia Abstract: Recent trends have revealed that SIP based IP-PBX DoS attacks contribute to most overall IP-PBX attacks which is resulting in loss of revenues and quality of service in telecommunication providers. IP-PBX face challenges in detecting and mitigating malicious traffic. In this research, Support Vector Machine (SVM) machine learning detection & prevention algorithm were developed to detect this type of attacks Two other techniques were benchmarked decision tree and Naïve Bayes. The training phase of the machine learning algorithm used proposed real-time training datasets benchmarked with two training datasets from CICIDS and NSL-KDD. Proposed real-time training dataset for SVM algorithm achieved highest detection rate of 99.13% while decision tree and Naïve Bayes has 93.28% & 86.41% of attack detection rate, respectively. For CICIDS dataset, SVM algorithm achieved highest detection rate of 76.47% while decision tree and Naïve Bayes has 63.71% & 41.58% of detection rate, respectively. Using NSL-KDD training dataset, SVM achieved 65.17%, while decision tree and Naïve Bayes has 51.96% & 38.26% of detection rate, respectively. The time taken by the algorithms to classify the attack is very important. SVM gives less time (2.9 minutes) for detecting attacks while decision tree and naïve Bayes gives 13.6 minutes 26.2 minutes, respectively. Proposed SVM algorithm achieved the lowest false negative value of (87 messages) while decision table and Naïve Bayes achieved false negative messages of 672 and 1359, respectively. Keywords: Voice over IP; Session Initiation Protocol; Attack; Security; Denial of Service, Support Vector Machine. 1. Introduction We have seen an increase in malicious attacks on the internet over the past few years as the internet continues to grow and integrate more facets of our everyday life than ever. These attacks are mostly targeted towards communications, payments, and many other aspects [1]. Therefore, the importance for network security professionals to effectively identify these different types of attacks and to prevent them from using various network security techniques runs constant. Voice over Internet Protocol (VoIP) is technology that uses connectivity over Internet Protocol (IP) networks to communicate with the system. In addition to traditional phone services including VOIP, it offers voice call flexibility and efficiency like that of the traditional Public Switched Telephone Network (PSTN). If we compare VoIP to traditional telephony, it has emerged as a standard for voice communication using the Internet and it allows the integration of more communication options and at lower cost compared to traditional telephony. A lot of interest has been devoted to strengthening the network of SIP without considering security of the protocol. SIP-based VOIP network can be prone to IP attacks. People should be aware of the different types of SIP attacks and countermeasures to overcome them. SIP based VoIP security issue has been met with a range of solutions and strategies. The relevance of a SIP-based VoIP communications security is well known among cyber security experts. There has been little research regarding DoS-based SIP security to this point in time. This paper is based on an assessment approach that depends on the use of a Real Network topology. To the authors’ knowledge, most of the publications reviewed do not use this type of deployment for evaluation. Testing DoS attacks on an operational network enables the most realistic testing environments. The second section presents Intrusion Detection Systems (IDS). The third section explains Anomaly detection techniques using Machine learning classifiers. Section 4 addresses the proposed machine learning classifier to combat attacks in SIP based VoIP followed by its performance metrics. This section also illustrates related work to benchmark the performance of proposed algorithm and Section 5 concludes the paper providing some pointers to future research work. 2. Intrusion Detection System (IDS) Denial of Service (DoS) attacks seek to make a server or system unavailable to its intended users [2]. An IDS is a system that monitors incoming and outgoing traffic to detect violations in the design. IDS could be a software or hardware system which detects malicious measures on computer systems to ensure system security. Intrusion can be characterized as any malicious behavior causing information system harm. Any attacks that may pose a threat to the privacy, credibility or availability of information would therefore be considered an intrusion. For instance, behavior that would prevent legitimate users from reacting to computer services is regarded as an intrusion. The right attack detection phase should be in a good defense system before any reaction. Any system to detect attacks is intended to detect intrusions before significant harm can occur. Any unauthorized attempt to view, disprove, alter, or damage information to make a network unsatisfactory is also called Intrusion [3].