RESEARCH ARTICLE Deployment and validation of out of band IP traceback approach (OBTA) in wireless mesh network Mouna Gassara 1 | Imen Bouabidi 1 | Faouzi Zarai 1 | Mohammad S. Obaidat 2 | Fellow of IEEE and Fellow of SCS 2 | Kuei‐Fang Hsiao 3 1 NTS'COM Research Unit, ENET'COM, University of Sfax, Tunisia 2 King Abdullah II School of Information Technology, University of Jordan, Amman Jordan and ECE Department, Nazarbayev University, Astana, Kazakhstan 3 Department of Information Management, Ming‐Chuan University, Taoyuan County 333, Taiwan Summary Wireless mesh networks (WMNs) have acquired recently enormous attention and momentum; therefore, security aspects have been a fundamental concern for them. Among catastrophic threats on WMNs, Denial‐of‐Service attacks that have become a severe danger because of their plug‐and‐play structural design. Unfortunately, preventing a Denial‐of‐Service attack presents a challenging issue. This fact is induced with the appearance of the source IP addresses spoofing. The resolution key of this issue is to reveal the attack source based on the path through which the attack packet passes. For this, many researchers in IP traceability field propose various methods and techniques to deal with the issue. In this article, we conceive a novel approach named out of band IP traceback approach in WMN (IEEE 802.11s). We create a new architecture using signaling messages for discovering the real source(s) of IP packets. Our solution is based on a security‐oriented signaling protocol. This protocol allows specialized signaling entities to communicate via reliable signaling information. This fact permits us to perform a simple and efficient traceback. In our novel approach, we use 2 radios: the first one transmits normal data packets whereas the second is reserved to exchange IP traceback information. The performance of the proposed scheme is analyzed via simulation analysis using the Network Simulator 3. The simulation results show that our scheme is efficient in dealing with the traceback problem in WMN environments. KEYWORDS WMNs, IEEE 802.11s, security, DoS attack, OBTA, signaling information, IP traceback, NS3 1 | INTRODUCTION Since their appearance, wireless mesh networks (WMNs) are considered a next generation pattern for the true wireless Internet. Because of the growing commercial attention in WMNs, IEEE formed a task group (IEEE 802.11s) for formal- izing the MAC and PHY standards. Nevertheless, IEEE 802.11s has not been ratified until the beginning of 2010. 1 The research field in WMNs concentrates on 1 side on routing aspect; security on other hand defines an important issue if not considered, then security breaches will threaten the whole function of WMNs. The security problem in these environments can be regarded as multidimensional problem including the security issue of mesh routers firstly and the security issue of mesh clients secondly. Received: 14 November 2017 Revised: 23 February 2018 Accepted: 4 March 2018 DOI: 10.1002/dac.3580 Int J Commun Syst. 2018;e3580. https://doi.org/10.1002/dac.3580 Copyright © 2018 John Wiley & Sons, Ltd. wileyonlinelibrary.com/journal/dac 1 of 18