1536-1276 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TWC.2018.2867180, IEEE Transactions on Wireless Communications 1 Unsupervised Machine Learning Based User Clustering in mmWave-NOMA Systems Jingjing Cui, Student Member, IEEE, Zhiguo Ding, Senior Member, IEEE, Pingzhi Fan, Fellow, IEEE and Naofal Al-Dhahir, Fellow, IEEE Abstract—Millimeter-wave non-orthogonal multiple access (mmWave-NOMA) systems exploit the power domain for multiple access to further enhance the spectral efficiency. User clustering and power allocation can effectively exploit the potential of NOMA in mmWave systems. This paper investigates the sum rate maximization problem of mmWave-NOMA systems under the constraints of the total transmission power and users’ pre- defined rate requirements. The formulated optimization problem is a non-linear programming problem and, thus, is non-convex and challenging to solve, especially when the number of users becomes large. Sparked by the correlation features of the users’ channels in mmWave-NOMA systems, we develop a K-means based machine learning algorithm for user clustering. Moreover, for a practical dynamic scenario where the new users keep arriving in a continuous fashion, we propose a K-means based on-line user clustering algorithm to reduce the computational complexity. Furthermore, to further enhance the performance of the proposed mmWave-NOMA system, we derive the optimal power allocation policy in a closed form by exploiting the successive decoding feature. Simulation results reveal that: 1) the proposed machine learning framework enhances the performance of mmWave-NOMA systems compared to the conventional user clustering algorithms; 2) the proposed K-means based on-line user clustering algorithm provides a comparable performance to the conventional K-means algorithm and strikes a good balance between performance and computational complexity. Index Terms—MmWave-NOMA, user clustering, machine learning, K-means. I. I NTRODUCTION With the tremendous traffic growth envisioned in fu- ture wireless networks, more bandwidth is a critical need. Millimeter-wave (mmWave) communications with large band- width is highly attractive for fifth generation (5G) wireless communications. In contrast to communications in the sub- 6 GHz frequency band, mmWave communications use the The work of Jingjing Cui and Pingzhi Fan was supported by the National Science Foundation of China (NSFC, No. 61731017), and the 111 Project (No. 111-2-14). The work of Z. Ding was supported by the UK EPSRC under grant number EP/N005597/1, NSFC under grant number 61728101 and H2020- MSCA-RISE-2015 under grant number 690750. The work of N. Al-Dhahir was made possible by NPRP grant ♯ 8-627-2-260 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. J. Cui and P. Fan are with the Institute of Mobile Communications, Southwest Jiaotong University, Chengdu 610031, P. R. China. (email: cui- jingj@foxmail.com, p.fan@ieee.org). Z. Ding is with the School of Electrical and Electronic Engineer- ing, The University of Manchester, Manchester, M13 9PL, UK. (e-mail: zhiguo.ding@manchester.ac.uk). N. Al-Dhahir is with the Department of Electrical Engineering, Uni- versity of Texas at Dallas, Richardson, TX 75080 USA. (e-mail: ald- hahir@utdallas.edu) Part of this work has been presented in IEEE Vehicular Technology Conference (VTC), Porto, Jun. 2018. frequencies between 30 and 300 GHz [1], which makes mmWave transmission highly directional. This feature ensures that the propagation in the mmWave spectrum experiences less interference and achieves a high data rate. Furthermore, due to the rapid growth of the number of mobile devices, applying new multiple access techniques to mmWave communications is essential to realize the mas- sive connectivity and further enhance the availability of the mmWave spectrum. As a promising 5G technology, non- orthogonal multiple access (NOMA) with power-domain mul- tiplexing provides an effective solution to improve the spectral efficiency. The key idea of NOMA is to serve multiple users at the same resource (e.g., time/frequency) by exploiting the user differences in the power domain. NOMA is motivated by the fact that the successive interference cancellation (SIC) tech- nique applied at the receivers exploits the channel differences across the users. This power domain feature provides rich opportunities for NOMA to support massive connectivity and meet the users’ diverse quality of service (QoS) requirements [2]. Motivated by its potential advantages, the application of NOMA to mmWave transmission has attracted considerable research interests [3–5]. The mmWave-NOMA system has the following characteristics: 1) highly directional transmissions in mmWave that can make the users’ channels strongly corre- lated, which facilitates an effective combination of NOMA and mmWave; 2) higher capacity and more connectivity achieved by exploiting the NOMA principle. These characteristics fur- ther increase the benefits of mmWave-NOMA systems. A. Prior Works In multiple-input and multiple-output (MIMO)-NOMA transmissions, it is desirable that one beam can support a cluster of users in mmWave-NOMA. In [3], a mmWave- NOMA transmission scheme was proposed to exploit the fact that the channels of the mmWave users in one cluster are highly correlated. However, [3] focused on the characterization of the sum rate and outage probabilities for mmWave-NOMA systems without considering the optimization of user clus- tering and power allocation. [4] investigated the potential of NOMA in mmWave massive MIMO systems by assuming that the users were pre-grouped. The capacity analysis of NOMA in mmWave massive MIMO systems was performed in [5] by assuming that the channels of the users from different groups are perfectly orthogonal. To further improve the performance of mmWave-NOMA systems, a joint design of user selection