Spectral Fractal Dimension Trajectory to Measure Cognitive Complexity of Malicious DNS Traffic Muhammad Salman Khan, Sana Siddiqui, Ken Ferens, and Witold Kinsner Dept. of Electrical and Computer Engineering, University of Manitoba, Winnipeg, MB, Canada muhammadsalman.khan@umanitoba.ca, siddiqu5@myumanitoba.ca, ken.ferens@umanitoba.ca, witold.kinsner@umanitoba.ca Abstract—Internet traffic exhibits long range dependence (persistence), scale invariance and self-similarity or self- affinity which are the known characteristics of fractals. Moreover, these characteristics of fractals can be extracted and quantified from an internet data time series using non-integer dimensions (fractal dimensions). The notion of cognitive complexity is also very well represented by the fractal dimensions, e.g., high value of fractal dimension of an object implies that the complexity of this object is higher than the one with lower fractal dimension. In addition, a multifractal object is more complex than a monofractal object and this can also be characterized to identify the degree of complexity. In this work, we have shown that the complexity introduced by distributed denial of service (DDoS) attack packets in DNS (Domain Name System) traffic is higher than the complexity of DNS traffic with no DDoS attack packets. A power spectrum density of the data series was used to calculate the spectral fractal dimension, and the performance of the proposed algorithm is validated using mathematical fractal Brownian motion process (fBm) and the real data sets. A sequence of spectral fractal dimension measurements of the time series (also known as a trajectory of spectral fractal dimension measurements or spectral fractal dimension trajectory (SFDT)) was generated to show the changing complexity of the series in time domain. Keywords—Denial of service, Domain Name System (DNS), cyber threats, complexity, multifractal, power spectrum density, time series, spectral fractal dimension trajectory (SFDT), variance fractal dimension trajectory, malicious traffic. I. INTRODUCTION Certain cyber attackers exploit the vulnerabilities of DNS (Domain Name System) protocol to disrupt DNS services using various methods. Distributed denial of service (DDoS) DNS amplification attack is one of such methods which uses legitimate DNS servers to piggy back and amplify the payload of DNS packets. There is no useful information contained in such packets and they reduce the available bandwidth of the network. The attack is launched by sending a broadcast message to the legitimate computer nodes after manipulating the source and destination IP addresses of the message such that the receiver nodes receive these messages from an authentic node which acts as a piggy back node for the attacker. The victim nodes receive the DNS traffic continuously from the DNS servers without generating any DNS request [1] [2]. Since many authentic servers send these DNS response packets to the victim node, continuously, the resulting persistent high rate of traffic overwhelms the victim node. The victim node becomes unable to process the packets received at the rate they are being sent, and this causes the victim node to loose/drop packets, including packets received from other legitimate sources. Consequently, the victim node is unable to process other legitimate network requests, thus resulting in a denial of DNS service of those legitimate network requests. Furthermore, the attacker cannot be traced because the attack is launched using authentic source nodes and the attacker remains anonymous. II. LITERATURE REVIEW In order to detect DNS denial of service attacks with high accuracy, it is required to devise a solution that can differentiate accurately between normal and anomalous packet flows. Signature based methods cannot accurately detect DNS attacks, because there is no known signatures of DNS attack packets that can be used to differentiate between normal and attack packets. In other words, DNS attack packets resemble authentic DNS packets. However, there are various methods in the literature, which attempt to detect DNS DDoS amplification attacks. The authors in [2] describe a method of mapping and monitoring the DNS 36 Int'l Conf. Security and Management | SAM'16 | ISBN: 1-60132-445-6, CSREA Press ©