RESEARCH ARTICLE A model-driven approach for experimental evaluation of intrusion detection systems Anas Abou El Kalam 1 * , Mohamed Gad El Rab 2 and Yves Deswarte 3 1 OSCARS, ENSA—Cadi Ayyad University, Marrakesh, Morocco 2 NIST, Cairo, Egypt 3 LAAS-CNRS, Toulouse University, Toulouse, France ABSTRACT Because attacks are becoming more frequent and more complex, intrusion detection systems (IDSes) need significant improvements to be able to detect new attacks and variants of already known attacks. It is thus necessary to assess precisely their quality of detection, performance, and robustness in the environment where they will be deployed. In this paper, we present an evaluation approach designed to overcome most of the identified weaknesses in several IDS evaluation: the lack of a rigorous methodology, the use of non-representative test datasets, and the use of inappropriate metrics. In our approach, model-based evaluation is combined with experimental testing. Because testing an IDS against all possible attacks is practically impossible, we propose a classification of elementary attacks and a model of attack processes. Then, we developed the attack planning and injection tool that helps security administrators to plan and select the most relevant attack scenarios. Attack planning and injection tool is able to generate and carry out concrete and adaptable attacks on specifically identified computers. To demonstrate the validity of our approach, we experimented our tool in a case study environment to compare well-known IDSes. Copyright © 2013 John Wiley & Sons, Ltd. KEYWORDS IDS; security evaluation; security testing *Correspondence Anas Abou El Kalam, OSCARS, ENSA—Cadi Ayyad University, Marrakesh, Morocco. E-mail: a.abouelkalam@uca.ma 1. INTRODUCTION The revolution of information and telecommunication technologies has fundamentally changed our society. All sectors are affected by the diffusion of these technologies that bring with them not only positive effects but also new risks. Some new attacks exceed all expectations, by their complexity, damage capacity, and propagation. Unfortunately, evidence shows that companies or even countries lack the necessary technology and human resources to control this new situation. Worse, even when several lines of defense are constructed, perfect security cannot be guaranteed. For this reason, intrusion detection systems (IDSes) are now a key component in securing any computing system or network. In fact, the philosophy behind intrusion detection is that even when several protection mechanisms are used, powerful attacks are likely enough for some intrusions to occur. The detection of intrusion occurrences is as important as protection because ignoring security breaches may result in a continuous leakage of information and thereby significant loss. Nevertheless, IDS implementations stay far below the expectations. One of our major aims is thus to fill a serious gap in the IDS evaluation field, by overcoming the limitations of existing evaluation techniques, and to construct unbiased datasets for IDS evaluation [1,2]. In particular, it is difficult to generate a good test dataset that is representative of real world attacks and at the same time flexible enough to be adapted to a specific deployment environment. The first objective of this work is thus to produce well-constructed, representative, manageable, and flexible attack datasets. Obviously, the number of threats and possible attacks is huge, and it would be a laborious task to launch as many tests as possible attacks. We thus propose to classify attacks. A good classification of attacks would, ideally, allow to constitute equivalence classes and then to test the IDS against only one or a small number of attacks of each class. This will greatly reduce the size of generated datasets. However, this is not enough in practice. Usually, to achieve their goals, attackers often carry out scenarios with a set of organized activities, including malicious activities and apparently normal activities. To understand the real attack activities, we need to analyze the different attack SECURITY AND COMMUNICATION NETWORKS Security Comm. Networks (2013) Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/sec.911 Copyright © 2013 John Wiley & Sons, Ltd.