Learning Dynamics Across Similar Spatiotemporally Evolving Systems Joshua E. Whitman Department of Mechanical Science and Engineering University of Illinois Champaign, IL 61801 jewhitm2@illinois.edu Girish Chowdhary Department of Agricultural and Biological Engineering University of Illinois Champaign, IL 61801 girishc@illinois.edu Abstract We present a machine learning model, which we term Evolving Gaussian Processes (E-GP), that can generalize over similar spatiotemporally evolving dynamical systems. We show that this differentially-constrained model can not only estimate the latent state of a large-scale distributed system evolving in both space and time, but that a single such model can generalize over multiple physically similar systems over a range of parameters using only a few training sets. This is demonstrated on computational flow dynamics (CFD) data sets of fluids flowing past a cylinder. Although these systems are governed by highly nonlinear partial differential equations (the Navier-Stokes equations), surprisingly, our results show that their major dynamical modes can be captured by a linear dynamical systems layered over the temporal evolution of the weights of stationary kernels. Furthermore, the models generated by this method provide easy access to physical insights into the system, unlike comparable methods like Recurrent Neural Networks (RNN). The low computational cost of this method suggests that it has the potential to enable machine learning approximations of complex physical phenomena for design and autonomy tasks. 1 Introduction One of the fundamental problems in the applied sciences is modeling large-scale stochastic phenomena with both spatial and temporal (spatiotemporal) evolution [1, 10], of which fluid dynamics is a prime example. The field of Computational Fluid Dynamics (CFD) is based on modeling fluid flow using a first-principles approach, solving the Navier-Stokes nonlinear partial differential equations using numerical methods. These simulations are resource-intensive, sometimes requiring days in a supercomputer to generate. This means they are ill-suited for design or machine-learning tasks that require access to dozens or hundreds of simulations of different but similar situations. They are even more poorly suited for online applications, such as autonomous aerial, ground, or water vehicles. In the machine learning and statistics communities, however, data-driven models of spatiotemporally evolving phenomena have been gaining more attention [6]. These models require no prior knowledge of the physics or configuration of the environment. If one could use this approach to generate highly efficient machine learning models of phenomena from a few examples instead of using costly physics simulations, the design and control of complex systems would be revolutionized. We have recently developed a new differentially-constrained hierarchic modeling method [13] termed Evolving GP, that layers a linear dynamic transition model on the weights of a kernel-based model. The advantage of E-GPs is that when the spatial and temporal dynamics are hierarchically separated and a linear transition is used, the learning problem becomes more tractable and complex spatiotemporal behaviors can still be captured. The main difference between E-GPs and existing methods in geostatistics [6, 15] is that we view the problem from a joint sensing and modeling viewpoint by constructing a Bayesian observer in a reproducing kernel Hilbert space. One major payoff of this perspective, is that it enables us to determine the optimal number and location of sensors for efficiently monitoring and predicting the state of the distributed system. Most importantly however, we have shown that E-GP models are generalizable across complex but parametrically similar PDEs. Our approach also leaves open the possibility of modeling nonlinear behavior in the weight-space evolution using neural networks as the transition models. 1.1 Related Work Kernel methods constitute a well-studied and powerful class of methods for inference in spatial domains [23], and have also found success in spatiotemporal modeling [6, 20]. Most recent techniques have focused on designing/learning nonstationary covariance kernels [11][8, 16, 17]. However, these approaches are nonconvex and require sophisticated optimization methods like MCMC. Furthermore, they scale poorly for large data sets like those generated by CFD. Within the CFD community literature, there has been recent interest in data-driven approximations of the Koopman operator [5], a linear but infinite-dimensional evolution operator on the observables of a system [26], by discovering the primary operator’s primary eigenfunctions, eigenvalues, and eigenmodes. Williams et al., recently integrated a popular approximation algorithm called DMD with the kernel trick, allowing the it to be extended to systems with much larger dimensions [25]. However, this method is restricted to approximating the operator and is only indirectly concerned with generative models, whereas our method is concerned with the evolution of the weights which can be directly used to compute observables. Most importantly, we demonstrate that our method can generalize across similar systems. To our knowledge, no neural network has been able to model large-scale CFD systems that evolve in both space and time. Recently, convolutional neural networks were used to model steady state flow profiles at low Reynolds numbers and at low resolution [9], but the