10590 IEEE TRANSACTIONS ONVEHICULAR TECHNOLOGY, VOL. 66, NO. 11, NOVEMBER 2017 Performance Analysis of Nonorthogonal Multiple Access for Downlink Networks With Antenna Selection Over Nakagami-m Fading Channels Yangyang Zhang , Jianhua Ge, Member, IEEE, and Erchin Serpedin , Fellow, IEEE Abstract—We investigate the system performance of a nonorthogonal multiple access (NOMA) based downlink amplify-and-forward relay net- work over Nakagami-m fading channels with imperfect channel state in- formation, where the base station and all users are provided with multiple antennas, while the relay is equipped with a single antenna. Two special conditions of interest (e.g., Nakagami-1, i.e., Rayleigh and Nakagami-2) are analyzed, and closed-form expression for the system outage probability is derived. Moreover, tight lower and upper bounds of the outage probabil- ity, and the outage probability in the high signal-to-interference-and-noise ratio regime, i.e., in the presence of error floor, which exists due to the channel estimation errors, are obtained. Finally, computer simulations are conducted to verify the accuracy of the numerical analysis and to confirm the superiority of the antenna selection and NOMA scheme. Index Terms—Antenna selection, imperfect channel state information, non-orthogonal multiple access, Nakagami-m fading, outage probability. I. INTRODUCTION Non-orthogonal multiple access (NOMA) has been considered as a promising signaling scheme in 5G wireless networks, due to its high radio-frequency spectrum efficiency and potential features to secure user fairness [1]. In [2], the outage performance was investigated in a hybrid relaying network, where NOMA as well as orthogonal multi- ple access (OMA) were utilized within user pairing groups and among each group, respectively. In [3], the study in [2] was extended to demon- strate the superior level of NOMA over OMA under two conditions, i.e., permanent power allocation and cognitive radio. More practical analog networks were considered in [4], where the performance of NOMA was compared with OMA and the superiority of NOMA scheme was con- firmed. In addition, in [5], the successive interference canceller (SIC) was used in a NOMA-based multiple-input multiple-output (MIMO) wireless system with a single user to analyze the system performance. However, so far, no in-depth study has been reported on the performance of multiple-antenna NOMA-based relaying networks over Nakagami-m fading channels. Reference [6] studied the system Manuscript received May 9, 2017; revised August 2, 2017; accepted Septem- ber 20, 2017. Date of publication September 26, 2017; date of current version November 10, 2017. This work was supported in part by the National Basic Research Program of China (973 Program) under Grant 2012CB316100, in part by the “111” project under Grant B08038, and in part by the National Natural Science Foundation of China under Grant 61501347. The review of this paper was coordinated by Dr. N.-D. D´ ao. (Corresponding author: Yangyang Zhang.) Y. Zhang and J. Ge are with the State Key Laboratory of Integrated Ser- vice Networks, Xidian University, Xi’an 710071 China (e-mail: zyy_xidian@ 126.com; jhge@xidian.edu.cn). E. Serpedin is with the Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77840, USA (e-mail: serpedin@ ece.tamu.edu). Digital Object Identifier 10.1109/TVT.2017.2756442 performance under Nakagami-m fading, yet, single antenna is unprac- tical in the actual networks. In this paper, 1 we pursue a performance analysis of a downlink multiple-antenna NOMA-based relaying net- work by considering independent and identically distributed (i.i.d.) Nakagami-m fading for its potential to capture the time-variations of propagation channel. The main contributions of this paper are summa- rized as follows: 1) A general multiple-antenna dual-hop relaying network with the actual channel estimation error is considered in this paper. De- tailed system performance analyses are conducted in terms of the outage probability. Closed-form expression of the system outage probability is derived. 2) To determine the influence on the design parameters, such as the degree of decline and the number of antennas, tight lower and upper bounds for the outage probability are derived and the diversity order is determined. 3) The value of error floor (EF) is computed. Simulation results corroborate the existence of EF and highlight the importance of considering imperfect channel state information (ICSI) in the analysis of the system performance. Throughout this paper, Pr(·) denotes probability, F Z (·) and f Z (·) symbolize the cumulative distribution function (CDF) and the probability density function (PDF) of a random variable Z , respectively. II. SYSTEM MODEL In this paper, we focus on a downlink relaying network, where a N B -antenna base station B and NN U -antenna mobile users U exchange information via the help of a single-antenna relay R. As described above, this model has its practical applications, such as in cellular communication systems and wireless sensor networks. Our paper mainly studies a homogeneous network topology, where all users are clustered relatively close together. As in the general circumstance, we assume that the direct path between B and U does not exist because of its long distance or the heavy shadow. All the nodes operate in a half-duplex mode and all channels undergo Nakagami-m fading with integer m. The transmit power at B and R are denoted by P B and P R , respectively. In the first phase, B will combine the coded symbol of N mobile users and transmit the summation signal x B to R using the best an- tenna that can maximize the SINR in R. The unit signal is expressed as x B = ∑ N n = 1 √ P B a n x n , where a n 2 and x n denote the power allo- cation coefficient and the coded signal of the n-th user, respectively, n = 1, 2, ··· N , a 1 2 ≥ a 2 2 ≥···≥ a N 2 and ∑ N n = 1 a n 2 = 1 is satis- fied. The received signal at R is given by y R = g max BR N  n = 1  P B a n x n + n R , (1) where n R ∼ CN (0,σ R 2 ) denotes the additive white Gaussian noise (AWGN) at R, and |g max BR | = max 1≤n B ≤N B |g n B BR | represents the real channel coefficient of the selected channel between B and R. Let 1 Considering lowering the system complexity and improving the realizable system capacity as design measures for the considered system model, two techniques of antenna selection are employed at the two source nodes as in Section II. 0018-9545 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.