INTERNATIONAL JOURNAL OF COMMUNICATION SYSTEMS Int. J. Commun. Syst. 2007; 20:1113–1130 Published online 15 November 2006 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/dac.858 Adaptive quality of service-based routing approaches: development of neuro-dynamic state-dependent reinforcement learning algorithms Abdelhamid Mellouk* ,y , Saı¨d Hoceı¨ni and Yacine Amirat LISSI/SCTIC IUT Creteil-Vitry University Paris XII, 122 rue Paul Armangot Vitry sur Seine 94400, France SUMMARY In this paper, we propose two adaptive routing algorithms based on reinforcement learning. In the first algorithm, we have used a neural network to approximate the reinforcement signal, allowing the learner to take into account various parameters such as local queue size, for distance estimation. Moreover, each router uses an online learning module to optimize the path in terms of average packet delivery time, by taking into account the waiting queue states of neighbouring routers. In the second algorithm, the exploration of paths is limited to N-best non-loop paths in terms of hops number (number of routers in a path), leading to a substantial reduction of convergence time. The performances of the proposed algorithms are evaluated experimentally with OPNET simulator for different levels of traffic’s load and compared with standard shortest-path and Q-routing algorithms. Our approach proves superior to classical algorithms and is able to route efficiently even when the network load varies in an irregular manner. We also tested our approach on a large network topology to proof its scalability and adaptability. Copyright # 2006 John Wiley & Sons, Ltd. Received 14 September 2005; Revised 22 June 2006; Accepted 7 September 2006 KEY WORDS: shortest-path routing; flow-based routing; N-best paths; neural networks; adaptive routing; neuro-dynamic state-dependent; reinforcement learning; traffic engineering 1. INTRODUCTION Internet has become the most important communication infrastructure of today’s human society. It enables the worldwide users (individual, group and organizational) to access and exchange remote information scattered over the world. Currently, due to the growing needs in telecommunications (VoD, video conference, VoIP, etc.) and the diversity of transported flows, Internet network does not meet the requirements of the future integrated service networks that *Correspondence to: Abdelhamid Mellouk, LISSI/SCTIC IUT Creteil-Vitry University Paris XII, 122 rue Paul Armangot Vitry sur Seine 94400, France. y E-mail: mellouk@univ-paris12.fr Copyright # 2006 John Wiley & Sons, Ltd.