Natarajan Meghanathan et al. (Eds) : NETCOM, NCS, WiMoNe, GRAPH-HOC, SPM, CSEIT - 2016 pp. 83– 90, 2016. © CS & IT-CSCP 2016 DOI : 10.5121/csit.2016.61508            Changyoung An, Byeongjae Kim, and Heung-Gyoon Ryu Department of Electronics Engineering, Chungbuk National University, Cheongju, Korea 361-763 acy890217@naver.com, bj5236@nate.com, and ecomm@cbu.ac.kr ABSTRACT Recently, new waveforms for the 5th generation cellular system have been studied in many ways. UFMC, FBMC (filter bank multi-carrier) and W-OFDM (window orthogonal frequency division multiplexing) waveforms are very strong candidates as a new waveform for 5G system. In this paper, we have evaluated the spectrum characteristic and BER performance of the waveforms under the effect of nonlinear HPA. Also, we like to show the comparison of the time- frequency resources of each system because it would be very important to estimate the spectral efficiency and communication throughput. As simulation results, it is confirmed that OOB power of each system increases, and OOB power increase of FBMC system is the biggest. Additionally, we have confirmed that performance of every system is degraded by strength of HPA nonlinearity, and every system needs the PAPR reduction method for the nonlinear distortion compensation and power saving, even though it would be more complicated. Comparison table for the time-frequency resources requirement for the each modulation systems is included. KEYWORDS new waveform; OFDM; FBMC; UFMC; HPA nonlinearity 1. INTRODUCTION The mobile traffic is being increased dramatically, because various mobile devices and multimedia services are being increased [1].Also, the growth of mobile traffic is being accelerated. It is difficult for the present mobile communication to support the mobile traffic required in the future [2].In order to solve the problem, studies for next generation 5G mobile communication has been carried actively [3-4]. Conventional orthogonal frequency division multiplexing (OFDM) based on multi-carrier has high-power out-of-band (OOB) [5]. This characteristic causes adjacent channel interference (ACI). OFDM uses a wide guard band in order to avoid ACI. It decreases spectral efficiency when a number of mobile devices simultaneously access a base station. Next generation mobile communication system requires high-level key performance indicators (KPIs). It is difficult for OFDM to satisfy the KPIs. Universal filtered multi-carrier (UFMC) and filter bank based multi- carrier (FBMC) are known as the candidate waveform for 5G mobile communication. When the