PROOF COPY [WR/2001/022478] 005306QWR PROOF COPY [WR/2001/022478] 005306QWR Neural Networks and Reinforcement Learning in Control of Water Systems B. Bhattacharya 1 ; A. H. Lobbrecht 2 ; and D. P. Solomatine 3 Abstract: In dynamic real-time control RTC of regional water systems, a multicriteria optimization problem has to be solved to determine the optimal control strategy. Nonlinear and/or dynamic programming based on simulation models can be used to find the solution, an approach being used in the Aquarius decision support system DSS developed in The Netherlands. However, the computation time required for complex models is often prohibitively long, and therefore such a model cannot be applied in RTC of water systems. In this study, Aquarius DSS is chosen as a reference model for building a controller using machine learning techniques such as artificial neural networks ANN and reinforcement learning RL, where RL is used to decrease the error of the ANN-based component. The model was tested with complex water systems in The Netherlands, and very good results were obtained. The general conclusion is that a controller, which has learned to replicate the optimal control strategy, can be used in RTC operations. DOI: 10.1061/ASCE0733-94962003129:61 CE Database subject headings: Water supply systems; Neural networks; Learning; Netherlands; Dynamic programs. Introduction Present-day water management typically considers the water sys- tem in its entirety, with the objective of creating and maintaining a sustainable living environment, taking into account all of the demands made on the water system by different interest groups. Because a water system is supposed to serve many different in- terests, ranging from flood control to recreation, water manage- ment is increasingly seen as a multicriteria problem in which these interests should be properly balanced. Optimal control ac- tions should be generated through regulating structures such as pumping stations, weirs, inlets, etc. to ensure proper management of the water system. In real-time control RTC, decision making is posed as a mul- ticriteria dynamic problem and the solution is found by solving an optimization problem. RTC alleviates the need for major invest- ments in infrastructure of existing and new water systems, and ensures a close match of the various time-varying requirements of different water uses, and may become an effective tool for water managers. In The Netherlands, much of the territory lies below mean sea level and it is the responsibility of the water boards of regional water systems to keep water levels down to a desirable level. This regional water system comprises low-lying polder areas, which are drained by regulating structures such as pumping stations, weirs, etc. to canals, which eventually discharge to the sea. At present, about 65 water boards are responsible for regional water control. The application of RTC in these water systems is being studied actively to make better use of existing resources. RTC may be applied as part of a physically based model that uses mathematical optimization techniques to solve the control prob- lem, but experience shows that such a model requires a long computing time, prohibiting its use in RTC for complex models. Recent advances in the field of machine learning ML suggest that it may complement the traditional techniques used in water system control. ML is an interdisciplinary subject, which is en- riched with concepts drawn from diverse fields such as statistics, artificial intelligence AI, information technology, biology, cog- nitive science, philosophy, control theory, and others. ML uses methods such as artificial neural networks ANNs fuzzy logic, decision trees, reinforcement learning RL, and other related methods Mitchell 1997, and is aimed at developing computer programs that can learn from experience or data and develop a decision-making capability. There are numerous successful appli- cations of ML, ranging from credit-card-credential analyses to robotics. The most popular ML method used in water engineering is ANNs. ANNs have been applied successfully in rainfall-runoff modeling Minns and Hall 1996; Dawson and Wilby 1998, res- ervoir optimization Solomatine and Torres 1996, stage- discharge relationships Bhattacharya and Solomatine 2000, and many other related problems. We used ANNs together with fuzzy-rule-based systems for building a controller for pumping in regional water systems Lobbrecht and Solomatine 1999. Sat- isfactory results were obtained in reproducing both the pumping strategy and the resulting water level Lobbrecht et al. 2000. An application of an ANN to the problem of efficiently pumping water out from low-lying polder areas in South Holland was in- 1 Research Fellow, Hydroinformatics, International Institute for Infra- structural, Hydraulic and Environmental Engineering, P.O. Box 3015, 2601 DA Delft, The Netherlands. E-mail: bha@ihe.nl 2 Senior Lecturer in Hydroinformatics, International Institute for Infra- structural, Hydraulic and Environmental Engineering, P.O. Box 3015, 2601 DA Delft, The Netherlands and HydroLogic BV, P.O. Box 2177, 3800 CD Amersfoort, The Netherlands. E-mail: ahl@ihe.nl 3 Associate Professor in Hydroinformatics, International Institute for Infrastructural, Hydraulic and Environmental Engineering, P.O. Box 3015, 2601 DA Delft, The Netherlands. E-mail: sol@ihe.nl Note. Discussion open until April 1, 2004. Separate discussions must be submitted for individual papers. To extend the closing date by one month, a written request must be filed with the ASCE Managing Editor. The manuscript for this paper was submitted for review and possible publication on December 5, 2001; approved on December 16, 2002. This paper is part of the Journal of Water Resources Planning and Manage- ment, Vol. 129, No. 6, November 1, 2003. ©ASCE, ISSN 0733-9496/ 2003/6-1– 8/$18.00. JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / NOVEMBER/DECEMBER 2003 / 1 PROOF COPY [WR/2001/022478] 005306QWR