Modeling an Anomaly-Based Intrusion Prevention System Using Game Theory El Mehdi Kandoussi ( ✉ ) , Iman El Mir, Mohamed Hanini, and Abdelkrim Haqiq Computer, Networks, Mobility and Modeling Laboratory, Faculty of Sciences and Technology, Hassan 1st University, Settat, Morocco kandoussi.elmehdi@gmail.com, iman.08.elmir@gmail.com, haninimohamed@gmail.com, ahaqiq@gmail.com Abstract. In Cloud Computing environment, the availability, authentication and integrity became a more challenging problem. Indeed, the classical solutions of security based on intrusion detection system and firewalls are easily bypassed by experienced attackers. In addition, the use of different technologies in term of security didn’t mitigate the attack considerably. To achieve network system’s security with the complexity and the diversity of attack types is too difficult and costly. However, to make them more resistant to attacks, anomaly-based Intrusion Prevention System (IPS) are used. Such systems take into consideration the prob‐ ability of legitimacy of a packet if it didn’t match any signature of malicious packets. In this paper, a competitive normal form game is developed based on the probability of packets’ legitimacy and the trust that an IPS has over the owner of the packet. Furthermore, a decision is made about dropping, accepting or testing packet in the network, and different Nash Equilibriums are calculated based on the system’s parameters. Our approach demonstrated its feasibility in term of prediction of the cases in which the system could be compromised and the actions that should be performed in case of an intrusion. Keywords: Cloud computing · Security · Anomaly-based IPS · Game theory Nash equilibrium 1 Introduction Cloud computing has recently emerged as a well evolved computer technology area. According to the National Institute of Standards and Technology (NIST) [1] introduces cloud computing as “a model for enabling convenient, on demand network access to a shared pool of configurable computing resources (e.g., network, servers, storage, appli‐ cations and services) that can be rapidly provisioned and released with minimal manage‐ ment effort or service provider interaction. In the last few years, Cloud Computing becomes more challenging in term of security. Furthermore, different attacks bypassed easily the static measures of security based on rules of a security policy or a signature database of malicious packets [2]. For this reason different solutions as IPS that are proactive are developed. These types of measures that not only alert in case of an attack but also suspect malicious behavior of © Springer International Publishing AG, part of Springer Nature 2018 A. Abraham et al. (Eds.): IBICA 2017, AISC 735, pp. 266–276, 2018. https://doi.org/10.1007/978-3-319-76354-5_24