FRMDN: Flow-based Recurrent Mixture Density Network Seyedeh Fatemeh Razavi, Reshad Hosseini razavi_f@ut.ac.ir, reshad.hosseini@ut.ac.ir University of Tehran Abstract Recurrent Mixture Density Networks (RMDNs) are a relativity old probabilistic model that are consisted of two main parts: a Recurrent Neural Network (RNN) and a Gaussian Mixture Model (GMM), in which a kind of RNN (almost LSTM) is used to find the parameters of a GMM in every time step. While available RMDNs have been faced with different difficulties. The most important of them is high-dimensional problems. Since estimating the covariance matrix for the high-dimensional problems is more difficult, due to existing correlation between dimensions and satisfying the positive definition condition. Consequently, the available methods have usually used RMDN for almost 3-dimensional problems with a full covariance matrix or for high-dimensional problems with a diagonal covariance matrix. One approach to tackle this problem is used in old speech recognition methods (based on GMM). Indeed, they used factorization and tying schemes to reduce the number of parameters, increase the number of components in the mixture, improve the power of modeling, and prevent overfitting. This approach has only used in GMM’s literature, while it could be helpful for RMDNs, too. Hence, in this paper with inspiring the mentioned approach, we consider a tied configuration for each precision matrix (inverse of the covariance matrix) in RMDN as (Σ −1 k = UD k U) to enrich GMM rather than considering a diagonal form for it. But due to simplicity, we assume U be an Identity matrix and D k is a specific diagonal matrix for k th component. Until now, we only have a diagonal matrix and it does not differ with available diagonal RMDNs; while we know a diagonal covariance matrix supposes independence among dimensions. Besides, Flow-based neural networks are a new group of generative models that are able to transform a distribution to a simpler distribution and vice versa, through a sequence of invertible functions. Therefore, we tried to apply this kind of precision matrix on transformed data. At every time step, the next observation, y t+1 , has been passed through a flow-based neural network to obtain a much simpler distribution. Finally, we applied a diagonal GMM on transformed observations. As a use case, we applied the proposed method for a Reinforcement Learning problem is known as the world model. Experimental results verify the superiority of the proposed method to the base-line method in terms of Negative Log-Likelihood (NLL) for RMDN and the cumulative reward for a controller with fewer population size. I. Introduction Generative models as an important field of unsupervised learning have tried to generate samples from a data distribution. They attempt to find a good representation of data distribu- tion to learn how to generate samples from that distribution. Moreover, designing a generative model with tractable likelihood and efficient sampling method has been considered as a challenging task in machine learning [1]. In other words, generative models try to estimate a distribution somewhat sample data have been drawn from it. Within high dimensional data, this task is more complicated, because it has usually been expressed like a joint probability distribution. Further, it is applicable in lots of different fields such as image generation [2], 1 arXiv:2008.02144v1 [cs.LG] 5 Aug 2020