Dynamic Selection of Wireless/Powerline Links using Markov Decision Processes Dacfey Dzung, Yvonne-Anne Pignolet ABB Corporate Research Baden-Dättwil, Switzerland dacfey.dzung@ch.abb.com, yvonne-anne.pignolet@ch.abb.com Abstract— Communication networks for smart grids may consist of a mixture of legacy and new links using heterogeneous technologies, such as copper wires, optical fibers, wireless and powerline communication. If nodes are connected by two or more links, such as wireless and powerline, the sender of a message must decide on which link to transmit the next message. This paper considers the problem of dynamically selecting the link, based on success/failure (acknowledgement) of previous transmissions. The novel method is based on Markov (Gilbert- Elliott) channel models of lossy and time varying links. It specifies how to employ success/failure observations to rank the links optimally, with the objective function to maximize throughput. The theory of partially observable Markov decision problems (POMDP) provides the basic framework. We compare this new method with known linear learning strategies. I. INTRODUCTION A. Heterogeneous Smart Grid Communication Networks A Smart Grid consists of a diverse set of devices such as SCADA controllers, Ring Main Units (RMUs), Remote Terminal Units (RTUs), smart meters and routers. Consequently, smart grid communication networks are likely to use a variety of communication technologies, among them wireless and powerline communication (PLC) [11]. The existence of more than one communication technology increases redundancy and reliability. For this purpose networking stacks supporting multiple communication technologies are necessary to build a unified network [12], with a routing protocol that takes the characteristics of different communication technologies into account. As an example, imagine a network of devices that are capable of communicating not only over wireless links, but also over PLC. When one link is unavailable or faulty, the other should be used. For efficiency and power reasons, messages should not be transmitted on all links in parallel. Thus, a suitable routing solution requires automatic, transparent switching between the available communication interfaces depending on the state of the links. B. Routing Protocol RPL and Link Metrics The IETF Working Group Routing over Low power and Lossy networks (ROLL) has proposed an IPv6 Routing Protocol for Low-power and Lossy Networks (RPL, RFC 6550) [8]. In RPL, the most frequently used link metrics are delay, expected transmission count (ETX), energy consumption, and received signal strength. These metrics are measured and updated dynamically, in order to adapt the routing decisions to the time varying behaviour of the underlying links. Time constants of the link metric measurements and of the routing adaptation affect each other and determine the overall routing performance, but there is no well-defined coordination between these two processes. Link metric measurements are usually updated by some simple averaging, e.g. using some first order smoothing filter. When no packet transmission has occurred on a link, the last available measurement is used for routing. In this paper we propose a new link metric update and link selection method. The new approach is to exploit the Markov properties of the underlying channel to derive an optimum method of updating the link metric and of selecting the best link, using the theory of Partially Observable Markov Decision Processes (POMDP). This is described in Section II and III. In order to assess this new method, we compare the new method with a known stochastic learning scheme in Section IV and V. II. COMMUNICATION CHANNELS A. Lossy and Time-Variant Channel Models Finite State Markov Channels (FSMC) are simple models of time-varying communication channels [1], where the state transition probabilities characterize the dynamic behaviour of the channel. The simplest FSMC model is the well-known Gilbert-Elliott (GE) channel which has only two states ‘good’ (G) and ‘bad’ (B), with transition probabilities ( ) G s G s t t = = = +1 1 Pr l ( ) G s B s t t = = = - +1 1 Pr 1 l ( ) B s G s t t = = = +1 0 Pr l , ( ) B s B s t t = = = - +1 0 Pr 1 l . IEEE SmartGridComm 2013 Symposium - Communication Networks for Smart Grids and Smart Metering 978-1-4577-1702-4/13/$31.00 ©2013 IEEE 277