Flexible Charging Optimization for Electric Vehicles using MDPs-based Online Algorithms Nikita V. Tomin, * Jonas Maasmann ** Alexandr B. Domyshev * * Melentiev Energy Systems Institute SB RAS, Irkutsk, 664033 Russia (e-mail: tomin.nv@gmail.com). ** Institute of Energy Systems, Energy Efficiency and Energy Economics, TU Dortmund, Dortmund, Germany, (e-mail: jonas.maasmann@tu-dortmund.de) Abstract: In the paper, we formulate the problem of charging electric vehicles with a time- dependent energy source as a Markov Decision Process (MDP), with states defining the presence of cars, their individual levels of charge as well as the level of available renewable energy and storage devices. We exploit MDPs-based online algorithms such as Monte-Carlo Tree Search (MCTS) to overcome the scalability issues associated with charging of a large number of EVs, which corresponds to real distributed networks with flexible options. Using MCTS, we were able to generate optimal policies that balanced the energy toll on the electric grid with the final charge levels of each vehicle. We compare the performance of offline MDP solvers (Discrete Value Iteration algorithm) and online MDP solvers (MCTS) as well as reinforcement learning-based solvers (Q-learning) to find the optimal policy for EV’s flexible charging optimization. Keywords: electric vehicles, charging, flexibility, Markov decision process, stochastic optimization 1. INTRODUCTION The large-scale integration of electric vehicles (EVs) into the power grid brings both challenges and opportunities to the system performance. On one hand, the load demand from EV charging imposes a large impact on the stability and efficiency of the power grid. On the other hand, EVs could potentially act as mobile energy storage systems to improve power grid performance, such as load flat- tening, fast frequency control, and facilitating renewable energy integration. Evidently, uncontrolled EV charging could lead to inefficient power network operation or even security. Since deep market penetration of EVs will impose substan- tial current demands on an already fragile electricity grid, an optimal policy is sought to schedule smooth charging of EVs overnight and minimize the need for non-renewable electricity sources. In the paper, we propose to do by structuring the problem as a Markov Decision Processes (MDP) and performing online MDP solvers 2. PROBLEM STATEMENT Especially in the distribution network, an uncontrolled integration of EVs causes an increased need for grid ex- ? This work was supported by the Russian Scientific Foundation (No. 19-49-04108) and the German Science Foundation/DFG (No. RE 2930/24) under the project ”Development of Innovative Tech- nologies and Tools for Flexibility Assessment and Enhancement of Future Power Systems”. pansion. Table 1 shows the estimated grid expansion costs of different studies for the German grid. They calculate costs between 11bn euro and 253bn euro for uncontrolled and uncoordinated charging of EV’s battery for different scenarios. Table 1. Estimated Grid Extension Cost Study Year Grid Level EV pene- tration Grid ex- tension costs Pregger and et al (2012) 2012 HV, MV, LV 5,1 Mio ≺ 3% Friedl and et al. (2018) 2018 LV 50%, 100% 11, 26 bn euro Brundlinger and et al. (2017) 2017 HV, MV, LV 100% 146-253 bn euro Uncontrolled charging processes, especially on private charging infrastructure, cause load profiles whose max- imum values are at the same time as the maximum of the standard load profile. Figure 1 shows the load profiles of the two charging cases “charging after arriving after the last trip” and “charging after arriving after work”. Both profiles are for charging at home and calculated for a maximum charging power of 3.7 kW and 11 kW J.Maasmann (2019). In combination with the standard load profile overload situations or voltage violations are the consequence. A temporal shift of the two load peaks can lead to a reduction of the cumulated total load and thus prevent overload Preprints of the 21st IFAC World Congress (Virtual) Berlin, Germany, July 12-17, 2020 Copyright lies with the authors 12793