XXXVI SIMPÓSIO BRASILEIRO DE TELECOMUNICAÇÕES E PROCESSAMENTO DE SINAIS - SBrT2018, 16-19 DE SETEMBRO DE 2018, CAMPINA GRANDE, PB Spectral Efficiency of Massive MIMO using FBMC-OQAM Modulation Felipe Kurpiel Jose, Luis Lolis, Eduardo Parente Ribeiro and Felipe Alex Pinto Abstract— This article covers the potential of Filter Bank Multicarrier (FBMC) modulation to be used in the future 5G wireless networks where Massive Multiple-Input Multiple-Output (MIMO) will be deployed. The study compares orthogonal fre- quency division multiplexing (OFDM) with FBMC. The former is the multiplexing technique in 4G communications and the latter is one of the strongest candidates to replace OFDM in 5G networks. This comparison evaluates the spectral efficiency (SE) of a massive MIMO (MM) system uplink under a single- cell environment. Due to the absence of the cyclic prefix, the FBMC has better SE than the OFDM as the signal-to-noise-ratio (SNR) increases. However, to the best of the authors’ knowledge, this study has not yet been conducted under a MM scenario. In summary, this article presents an analysis of SE of FBMC considering a MM setup. While limiting the modulation to 64- Quadrature Amplitude Modulation (QAM) per sub carrier, it was observed that as the SE increases, the required number of antennas for the OFDM becomes the double or the triple of the counterpart using FBMC, or even it is not achieved by the OFDM. Keywords— Massive MIMO, FBMC, M-QAM modulation, spectral efficiency, 5G I. I NTRODUCTION The traffic of wireless communication networks has grown exponentially and transmission rates are nearing 1 Gb/s nowa- days, which leads to higher demands on system capacity. Mo- reover, designing wireless links with superior speed, quality- of-service and capability represents a significant engineering and research challenge. [1] Multiple-input multiple-output (MIMO) systems have emer- ged to serve tens of user equipment (UE) by employing hundreds of base station (BS) antennas in the same time- frequency resource. As a definition, Massive MIMO (MM) is a multi-user MIMO technology where a number K of UE antennas are serviced on the same time-frequency resource by a BS with M antennas such that M >> K. In reality, MM has become the strongest candidate to increase the capacity of multiuser networks [2]. Many studies analyses the spectral efficiency (SE) of MM under different scenarios [3]–[5]. In all of them, the orthogonal frequency division multiplexing (OFDM) is adopted as the multiplexing / modulation scheme. It is important to point out that all these studies considered that each sub-carrier has a Gaussian distributed modulation, being that a condition to achieve the Shannon limit per channel [6]. Felipe Kurpiel Jose, Luis Lolis, Eduardo Parente Ribeiro and Felipe Alex Pinto. Federal University of Parana (UFPR), Curitiba-PR, Brazil, E- mails: lipekurpiel@gmail.com, luis.lolis@eletrica.ufpr.br, edu@ufpr.br, fe- lipe.fap.ufpr@gmail.com. This work was partially supported by UFPR. As explained in [7], every existent waveform have its pros and cons, hence the benefits of large antenna arrays can turn a MM specific waveform combination more attractive than others. Compared to existing 4G technologies, 5G is targeting much higher throughput with sub-ms latency and utilizing higher carrier frequencies and wider bandwidths [8]. Given that, the filter bank multicarrier (FBMC) represents a possibility to provide higher SE and it is much more suited to a potential 5G system than OFDM [9]. In [10], the OFDM and FBMC SEs are compared with the theoretical bounds for Rayleigh fading and a single-input single-output (SISO) transmission. Each sub-carrier changed between M-QAM going from M =4 to M = 64. In addition, block error-correcting codes having rates from 78/1024 to 948/1024 were also applied. The SNR for a specific throughput is achieved when the bit error rate (BER) around 10 -3 , which is obtained through Monte-Carlo simulations. This paper presents an SE analysis for the OFDM and FBMC modulation / multiplexing schemes for MM cells, combining the work in [10], [3] and [4]. The individual SE per SNR from [10] is applied in the averaged perceived SNR per user obtained in [3] and [4] in a MM cell, to evaluate the SE in the entire cell, with different number of antennas and cell size. By adopting this strategy, the modulations being considered in a single-cell environment on a MM system uplink are presented in a more realistic format. In other words, the FBMC and OFDM sub-carriers are not seen as Gaussian distributions. In summary, this approach provides a more realistic scenario for the analysis of SE in a MM system uplink than what is presented on both [3] and [4]. The remainder of this paper is organized as follows: Section II provides the theoretical framework for MM and FBMC. In Section III the method to analyze non-Gaussian modulations is described. In Section IV the SE for FBMC and OFDM in MM is presented. In Section V the numerical results are provided and insightful discussions are drawn. Finally, Section VI concludes the paper. II. THEORETICAL FRAMEWORK A. Uplink Massive MIMO The design and analysis of MIMO systems includes one BS equipped with an array of M antennas that receive data from K single-antenna users. In [3] the SE of a MIMO system uplink for a single cell environment is evaluated considering lower capacity bounds in a channel model that includes small-scale fading. The study focus on three different linear detectors at the BS: maximum-ratio combining (MRC), zero-forcing (ZF)