ResearchArticle A Novel Machine Learning-Based Approach for Security Analysis of Authentication and Key Agreement Protocols Behnam Zahednejad , Lishan Ke , and Jing Li Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangzhou, China Correspondence should be addressed to Lishan Ke; kelishan@gzhu.edu.cn Received 2 May 2020; Revised 29 July 2020; Accepted 25 September 2020; Published 16 October 2020 Academic Editor: Xiaolong Xu Copyright © 2020 Behnam Zahednejad et al. is is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. e application of machine learning in the security analysis of authentication and key agreement protocol was first launched by Ma et al. in 2018. Although they received remarkable results with an accuracy of 72% for the first time, their analysis is limited to replay attack and key confirmation attack. In addition, their suggested framework is based on a multiclassification problem in which every protocol or dataset instance is either secure or prone to a security attack such as replay attack, key confirmation, or other attacks. In this paper, we show that multiclassification is not an appropriate framework for such analysis, since au- thentication protocols may suffer different attacks simultaneously. Furthermore, we consider more security properties and attacks to analyze protocols against. ese properties include strong authentication and Unknown Key Share (UKS) attack, key freshness, key authentication, and password guessing attack. In addition, we propose a much more efficient dataset construction model using a tenth number of features, which improves the solving speed to a large extent. e results indicate that our proposed model outperforms the previous models by at least 10–20 percent in all of the machine learning solving algorithms such that upper- bound performance reaches an accuracy of over 80% in the analysis of all security properties and attacks. Despite the previous models, the classification accuracy of our proposed dataset construction model rises in a rational manner along with the increase of the dataset size. 1. Introduction Security protocols (cryptographic protocols) are widely used to transport application-level data in a secure manner. ese protocols usually apply a sequence of cryptographic prim- itives such as (a)symmetric encryption, digital signature, and hash function. e most important goals of security pro- tocols include key agreement or establishment, entity au- thentication, message authentication, and nonreputation [1]. For instance, Transport Layer Security (TLS) [2] is a well- known cryptographic protocol that is used to provide secure web connections (HTTPS). To prove the correctness of security protocols, various methods were developed over the last decades. ese methods can be divided into two main categories. Model-checking methods refer to the set of automated tools and methods that try to find attacks which violate security goals, rather than proving their correctness. ProVerif [3], Scyther [4], AVISPA [5], CryptoVerif [6], and so on are among the most well-known tools. eorem-proving methods are less automated methods that consider all possible protocol behavior to check whether the security goal is achieved or not. Although they cannot give a security attack, they provide a proof of the correctness of the protocol. BAN logic [7], Dolev-Yao model [8], and strand space [9] are examples of these methods. 1.1.MotivationandGoalofisPaper. e goal of this paper is to develop a novel machine learning-based protocol analysis scheme with much better efficiency that can dis- cover more security attacks and vulnerabilities. Previously, the application of machine learning in security analysis has been mainly limited to side-channel attack [10, 11] and symmetric cryptoanalysis [12, 13]. Our motivation for Hindawi Security and Communication Networks Volume 2020, Article ID 8848389, 15 pages https://doi.org/10.1155/2020/8848389