2162-2337 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LWC.2019.2912378, IEEE Wireless Communications Letters 1 Deep Learning Based Channel Estimation for Massive MIMO Systems Chang-Jae Chun, Jae-Mo Kang, Member, IEEE, and Il-Min Kim, Senior Member, IEEE Abstract—In this paper, we propose a deep learning (DL) based channel estimation scheme for the massive multiple-input multiple-output (MIMO) system. Unlike existing studies, we develop the channel estimation scheme for the case that the pilot length is smaller than the number of transmit antennas. The proposed scheme takes a two-stage estimation process: a DL-based pilot-aided channel estimation and a DL-based data- aided channel estimation. In the first stage, the pilot itself and the channel estimator are jointly designed by using both a two-layer neural network (TNN) and a deep neural network (DNN). In the second stage, the accuracy of channel estimation is further enhanced by using another DNN in an iterative manner. The simulation results demonstrate that the proposed channel estimation scheme has much better performance than the conventional channel estimation scheme. We also derive an useful insight into the optimal pilot length given the number of transmit antennas. Index Terms—Channel estimation, deep learning, massive MIMO system. I. I NTRODUCTION Multiple-input multiple-output (MIMO) with large-scale an- tenna arrays, so-called the massive MIMO, is one of the most promising techniques to increase the data rate and to maintain the high communication reliability for future wireless systems [1], [2]. In the massive MIMO system, a large scale antenna array is deployed typically at the base station (BS) to provide a considerable antenna gain, and this antenna gain highly depends on the channel estimation accuracy. Thus, obtaining accurate channel estimation is very important to ensure such benefits of the massive MIMO technique. In the literature, the issue of channel estimation has been studied for the massive MIMO systems, typically based on the linear minimum mean square error (LMMSE) method, e.g., [3]–[5]. However, the common assumption in all the literature including [3]–[5] is that the pilot length L s is equal to or larger than the number of transmit antennas N t , which is (very) large in the downlink of massive MIMO. Without this assumption of L s ≥ N t , the channel estimation performance is substantially degraded, as demonstrated in [6] for the LMMSE channel estimator. In the massive MIMO system, however, it is hard to justify the assumption of L s ≥ N t for three main reasons. First, to ensure L s ≥ N t , substantial amount of time resource should be used for pilot transmission, which results in much reduced resource for data transmission, leading to (very) low spectral efficiency. Second, the computational complexity required for channel estimation also grows as L s increases. Last, but not least, ensuring L s ≥ N t might be even This work was supported by the Natural Sciences and Engineering Research Council of Canada Discovery under Grant 2019-04727. C.-J. Chun and I.-M. Kim are with the Department of Electrical and Computer Engineering, Queen’s University, Kingston, ON K7L 3N6, Canada (e-mail: changjae.chun@queensu.ca; ilmin.kim@queensu.ca). J.-M. Kang is with the School of Intelligent Mechatronics Engineering, Sejong University, Seoul 05006, South Korea (e-mail: jaemo.kang@sejong.ac.kr). impossible at all, because L s cannot be larger than the channel coherence interval, which is usually uncontrollable. Therefore, in the massive MIMO system, developing a channel estimation scheme for L s < N t is one of the most important, yet very challenging, issues. In spite of the importance of channel estimation for L s < N t in the massive MIMO system, there was no such study in the literature. Unfortunately, developing an effective chan- nel estimator for L s < N t is a very difficult problem for the following reasons. In all the existing works assuming L s ≥ N t , the pilot matrix S was simply designed such that its row vectors were orthogonal, which was needed to avoid the interference among the pilot sequences transmitted from different antennas. When L s < N t , however, it is not possible to design the pilot matrix S in that way. 1 This means that the channel estimation problem has to be jointly solved with the additional problem of designing the pilot S, which turns out be very challenging because the optimal (or high performing) structure of S is unknown when L s < N t . Another issue is that, when L s < N t , there is no guarantee that the LMMSE is an optimal channel estimator. 2 This means that, in the design of channel estimator, one should not simply restrict the channel estimator to be linear, which implies that the problem at hand is a nonlinear (and typically nonconvex) optimization problem. Overall, individual (separate) problems of the pilot design and the channel estimation are indeed very difficult, not to mention the problem of their joint optimization, which in fact needs to be solved eventually. To the best of our knowledge, there has been no work in the literature to jointly design the pilot and the channel estimator for the range of L s < N t in the massive MIMO. This motivated our work. In this paper, to circumvent the challenges discussed above and to construct a very effective channel estimation mechanism (including pilot design and channel estimation) for the massive MIMO when L s < N t , we take a fundamentally different approach by proposing a deep learning (DL) based two- stage channel estimation scheme. In the first stage, the joint design of pilot and channel estimator is performed to minimize the mean square error (MSE) of channel estimation by con- structing both a two-layer neural network (TNN) and a deep neural network (DNN). In the second stage, to further enhance the channel estimation performance, we adopt an iterative channel estimation technique by constructing another DNN as an additional channel estimator, in which data detection and channel estimation are iteratively performed to improve the channel estimation performance. The simulation results 1 If L s ≥ N t is assumed as in the literature, it is straightforward to design the pilot matrix S of size N t × L s such that SS H = I Nt . However, when L s < N t , it is not possible to have SS H = I Nt . 2 When L s ≥ N t in the presence of additive white Gaussian noise, it is known that LMMSE is an optimal channel estimator [7].