Neighbor Discovery Protocol Anomaly Detection Using Finite State Machine and Strict Anomaly Detection Firas Najjar, Mohammad Kadhum, Member,IEEE, Homam El-Taj Abstract-Neighbor Discovery Protocol (NDP) is a stateless protocol used by Internet Protocol Version 6 (IPv6) to find hosts and routers in an IPv6 network. Lacking of authentication process makes NDP exposed to various attacks. Securing NDP is a critical task since the large deployment of the Internet is done in public areas such as airports, where no trust between users is existed. Many solutions were proposed to secure the NDP; however, most of them violate the design principle of NDP in terms of complexity and overhead. Hence, further research on NDP is needed in order to identify and model the points that would help improve NDP while reducing complexity. This research uses finite state machines (FSM) and Extended Finite State Machine (EFSM) to model the main mechanisms used by NDP for detecting NDP anomalies based on Strict Anomaly Detection. These models can be used as a network security tool or as research tool to study and investigate the behavior of NDP behavior. Index Terms—Anomaly Detection, Finite State Machine, Neighbor Discovery Protocol, Strict Anomaly Detection I. INTRODUCTION Internet Protocol Version 6 (IPv6) [1] was deployed to overcome IPv4 address exhaustion limitation. Furthermore, IPv6 intended to replace IPv4 which still carries the vast majority of Internet traffic in 2015. In July 2015, the percentage of users reaching Google services over IPv6 surpassed 7% [2]. IPv6 uses NDP [3] to perform a variety of link operations, such as finding routers on the link, address resolution for nodes (hosts and routers), and keep track of reachability state for other nodes. NDP includes IPSec in its original design; however, the documented literature, such as RFCs, do not give detailed instructions on how the use of IPsec. IPSec can only be used with a manual configuration of security associations due to bootstrapping problems when using Internet Key Exchange. As consequence, main enterprise companies and organizations were driven to deploy IPv6 and develop security polices to overcome the security issues in IPv6 [4-7]. Even IPv6 design includes the original specification of IPsec to secure IPv6, manual configuration is not recommended [8] and limits the applicability. Moreover, the designers of IPv6 suppose that the local area network consists of only trusted users; however, Eernst and Young [9], which it’s a global leader company in assurance, tax, transaction and advisory services, mentioned in their survey in 2013 and 2014 that “employees were seen as the most likely source of an attack, and still seen as a significant risk”. This research models NDP mechanism using FSM [10], which is a powerful modeling tool used to describe systems behavior. Furthermore, FSM can be used to detect abnormal behavior in NDP by describing the sequence of ICMPv6 messages between neighbors. However, NDP specification includes variable and operations which FSM is not powerful to model them, hence, the proposed mechanism extend FSM with operation and variables to describe and analyze NDP using EFSM modeling technique. Certainly, abnormal behavior generates additional amount of traffic or unauthorized sequence of packets. Consequently, modeling normal NDP behavior leads to detect these anomalies. By using Strict Anomaly Detection [11], which involves defining a set of rules for permitting events that reflect the normal behavior of NDP mechanism, allows detecting any activities that violate such rules. The rest of this paper is organized as follow: Section 2 presents background of NDP. Section 3 covers some related work. Section 4 models the normal behavior of NDP. Section 5 describes NDP anomaly detection method. Finally, the conclusion is presented in section 6. II. BACKGROUND IPv6 uses NDP to process Address Resolution, Router Discovery, Duplicate Address Detection (DAD), Neighbor Unreachability Detection (NUD), and Redirect traffic. To complete these tasks, NDP uses five ICMPv6 messages, which are: • Router Advertisement (RA) messages are originated by routers and sent periodicity, or sent in response to Router Solicitation to advertise their presence and send specific parameters such as MTU, Router Prefix, list of prefix and hop limits. Furthermore, Router advertisement contains Firas Najjar is with the National Advanced IPv6 Centre of excellence (NAv6), Universiti Sains Malaysia (USM),Pulau Pinang, Malaysia 11800, (e-mail:Firas@nav6.usm.my). Mohammad Kadhum is with the National Advanced IPv6 Centre of excellence (NAv6), Universiti Sains Malaysia (USM),Pulau Pinang, Malaysia 11800, (e-mail:Kadhum@nav6.usm.my) , and is with Telecommunications Research Lab, School of Computing, Queen’s University, Kingston, ON, Canada K7L 3N6, (e- mail:kadhum@cs.queensu.ca). Homam El-Taj is with Community college, Tabuk University, Tabuk, Saudi Arabia, (e-mail:h.eltaj@ut.edu.sa).