0090-6778 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TCOMM.2018.2876867, IEEE Transactions on Communications IEEE TRANSACTIONS ON COMMUNICATIONS 1 Error Probability Analysis of Non-Orthogonal Multiple Access over Nakagami-m Fading Channels Lina Bariah, Student Member, IEEE, Sami Muhaidat, Senior Member, IEEE, and Arafat Al-Dweik, Senior Member, IEEE Abstract—Non-orthogonal multiple access (NOMA) is cur- rently considered as a promising technology for the next genera- tion wireless networks. In this paper, the error rate performance of NOMA systems is investigated over Nakagami-m fading channels, while considering imperfect successive interference cancellation (SIC). In particular, the paper focuses on the pairwise error probability (PEP) analysis, where exact PEP expressions are derived to characterize the performance of all users under different fading conditions. The obtained PEP expressions are then used to derive an exact union bound on the BER. Through the derived PEP expressions, the asymptotic PEP analysis is presented to investigate the maximum achievable diversity gain of NOMA users. Moreover, using the derived BER bound, the power allocation problem for all users in NOMA systems is considered under average power and users BER constraints, which allows realizing the full potential of NOMA. Monte Carlo simulation and numerical results are presented to corroborate the derived analytical expressions and give valuable insights into the error rate performance of each user and the achievable diversity gain. Index Terms: Diversity gain, interference cancellation, Nakagami-m, NOMA, pairwise error probability. I. I NTRODUCTION T HE explosive growth of high data rate services and the proliferation of the Internet of Things (IoT) devices pose unique challenges for the next generation wireless communi- cations, among which are the high energy efficiency, spectral efficiency and massive connectivity. Non-orthogonal multiple access (NOMA) has recently emerged as a promising solution for the increasing demand for capacity in the upcoming fifth generation (5G) wireless communications. Different from con- ventional orthogonal multiple access (OMA) techniques that rely on orthogonal allocation of resources to eliminate multi- user interference, such as frequency division multiple access (FDMA), time division multiple access (TDMA) and code division multiple access (CDMA), the key principle of NOMA systems is to allow multiple users to share the same frequency The authors are with the Department of Electrical and Computer Engineering, Khalifa University, Abu Dhabi, UAE, E-mails:{lina.bariah, sami.muhaidat, arafat.dweik}@ku.ac.ae. S. Muhaidat is also with the Center on Cyber-Physical Systems, Khalifa University, Abu Dhabi, UAE. E e-mail: muhaidat@ieee.org A. Al-Dweik is also with the Department of Electrical and Computer Engineering, Western University, London, ON N6A 3K7, Canada. E-mail: dweik@fulbrightmail.org. This work was supported by ICT fund grant No. 11/15/TRA-ICTFund/KU. This work is published in part at the IEEE International Conference on Communications (ICC), 2018 [1]. and time resources [2]. NOMA systems permit a controlled level of interference from other users, by allocating different power levels to different users [3]. Low latency, spectral effi- ciency and connectivity are the key factors that stimulated the rise of NOMA systems, to meet the main requirements of the upcoming 5G wireless communications [4]. Nevertheless, this comes at the expense of increased receiver complexity, where successive interference cancellation (SIC) should be employed to eliminate the interference from other users’ signals before the intended users can detect their own signals. User fairness in NOMA systems is realized by allocating different power levels to different users, based on their channel conditions. In particular, users with weak channels are allocated high power coefficients, while users with stronger channels are allocated lower power coefficients. Although such allocation maintains user fairness, the performance of weak users is relatively poor, which is considered as a performance limiting factor in many scenarios due to error propagation [5]. For enhanced performance for all users, multiple antenna deployment in NOMA systems has been recently considered in the literature, where it is shown that using multiple antennas with the aid of a proper beamforming technique, can provide a remarkable enhancement in the system’s throughput and enable NOMA systems to meet the diverse quality of service (QoS) require- ments of all users [6]–[8]. Extensive research efforts have been conducted to inves- tigate the performance of NOMA systems from different perspectives and under different scenarios. For example, the outage probability and ergodic sum rate of NOMA systems over Rayleigh fading channels are analyzed in [9] for ran- domly deployed users, to characterize the impact of path loss. The reported results show that NOMA can enhance the ergodic sum rate dramatically; however, the outage probability performance depends on the user targeted data rate and power level. As an extension to [9], the authors in [10] analyzed the sum rate and ergodic sum rate for multiple-input multiple- output (MIMO) NOMA system. Exact and asymptotic outage probability analysis are derived in [11] for a hybrid relay- assisted multiple access system, where users are divided into groups and all users in each group implement NOMA, while users from different groups use OMA. Performance analysis of NOMA systems is investigated from user fairness stand- point for instantaneous and average channel state information (CSI) in [12], where different power allocation schemes are investigated to ensure user fairness. It is worth noting that