mathematics Review Reinforcement Learning Approaches to Optimal Market Making Bruno Gašperov 1, * , Stjepan Beguši´ c 1 , Petra Posedel Šimovi´ c 2 and Zvonko Kostanjˇ car 1   Citation: Gašperov, B.; Beguši´ c, S.; Posedel Šimovi´ c, P.; Kostanjˇ car, Z. Reinforcement Learning Approaches to Optimal Market Making. Mathematics 2021, 9, 2689. https://doi.org/10.3390/math9212689 Academic Editor: José Niño-Mora Received: 6 October 2021 Accepted: 18 October 2021 Published: 22 October 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affil- iations. Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). 1 Laboratory for Financial and Risk Analytics, Faculty of Electrical Engineering and Computing, University of Zagreb, 10000 Zagreb, Croatia; stjepan.begusic@fer.hr (S.B.); zvonko.kostanjcar@fer.hr (Z.K.) 2 Department of Informatics and Mathematics, Faculty of Agriculture, University of Zagreb, 10000 Zagreb, Croatia; pposedel@agr.hr * Correspondence: bruno.gasperov@fer.hr Abstract: Market making is the process whereby a market participant, called a market maker, simultaneously and repeatedly posts limit orders on both sides of the limit order book of a security in order to both provide liquidity and generate profit. Optimal market making entails dynamic adjustment of bid and ask prices in response to the market maker’s current inventory level and market conditions with the goal of maximizing a risk-adjusted return measure. This problem is naturally framed as a Markov decision process, a discrete-time stochastic (inventory) control process. Reinforcement learning, a class of techniques based on learning from observations and used for solving Markov decision processes, lends itself particularly well to it. Recent years have seen a very strong uptick in the popularity of such techniques in the field, fueled in part by a series of successes of deep reinforcement learning in other domains. The primary goal of this paper is to provide a comprehensive and up-to-date overview of the current state-of-the-art applications of (deep) reinforcement learning focused on optimal market making. The analysis indicated that reinforcement learning techniques provide superior performance in terms of the risk-adjusted return over more standard market making strategies, typically derived from analytical models. Keywords: deep reinforcement learning; reinforcement learning; finance; market making; machine learning; deep learning; survey; literature review 1. Introduction Modern financial markets are increasingly order-driven and electronic, with the elec- tronic limit order book (LOB) becoming the dominant trading form for multiple asset classes. This electronification of financial markets has led to the increased importance of certain (algorithmic) trading strategies, in particular market making strategies. Market making (MM) is the process whereby a market participant simultaneously and repeatedly posts limit orders on both sides of the limit order book of a given security, with the goal of capturing the difference between their prices, known as the quoted spread. A limit order book (LOB) is simply a collection of outstanding offers to buy or sell (limit orders). Traders that rely on MM strategies are simply referred to as market makers. A market maker might, for example, post a buy limit order at USD 99 and a sell limit order at USD 101. If both orders become executed (i.e., if both a counterparty willing to sell at USD 99 and a counterparty willing to buy at USD 101 emerge), the market maker will capture the spread, i.e., earn USD 2, all while providing the market with liquidity. However, if only one of the limit orders becomes executed, the market maker will not only fail to capture the spread, but also obtain a nonzero inventory and consequently take on the inventory risk that stems from fluctuations in the market value of the held asset. A risk-averse market maker hence adjusts its quotes to increase the probability of selling when its inventory is strictly positive, and vice versa, in order to dispose of the inventory and mitigate the associated risk. Market makers act to maximize their risk-adjusted returns while at the same time playing a key role in the market by providing it with liquidity. Optimal MM entails Mathematics 2021, 9, 2689. https://doi.org/10.3390/math9212689 https://www.mdpi.com/journal/mathematics