1536-1276 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TWC.2015.2412119, IEEE Transactions on Wireless Communications IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, MONTH 2015 1 A Bayesian Approach for Nonlinear Equalization and Signal Detection in Millimeter-wave Communications Bin Li, Chenling Zhao, Mengwei Sun, Haijun Zhang, Zheng Zhou Member, IEEE and Arumugam Nallanathan Senior Member, IEEE Abstract—For the emerging 5G millimeter-wave communica- tions, the nonlinearity is inevitable due to RF power amplifiers of the enormous bandwidth operating in extremely high frequency, which, in collusion with frequency-selective propagations, may pose great challenges to signal detections. In contrast to classical schemes which calibrate nonlinear distortions in transmitters, we suggest a nonlinear equalization algorithm, with which the mul- tipath channel and unknown symbols contaminated by nonlinear distortions and multipath interferences are estimated in receiver- ends. Attributed to the nonlinearity and marginal integration, the involved posterior density is analytically intractable and, unfortunately, most existing linear equalization schemes may be- come invalid. To solve this problem, the Monte-Carlo sequential importance sampling based particle filtering is suggested, and the non-analytical distribution is approximated numerically by a group of random measures with the evolving probability-mass. By applying the Taylor’s series expansion technique, a local- linearization observation model is further constructed to facilitate the practical design of a sequential detector. Thus, the unknown symbols are detected recursively as new observations arrive. Simulation results validate the proposed joint detection scheme. By excluding transmitting pre-distortion with high complexity, the presented algorithm is specially designed for the receiver-end, which provides a promising framework to nonlinear equalization and signal detection in millimeter-wave communications. Index Terms—5G Millimeter-wave communications, nonlin- ear equalization, signal detection, nonlinear power amplifier, Bayesian recursive approach, particle filtering I. I NTRODUCTION A TTRIBUTED to the availability of abundant unautho- rized spectrum resources in millimeter-wave (mm-Wave) frequency band, wireless personal communication networks (WPANs) and wireless local area networks (WLANs) operat- ing in such frequencies (e.g., 60GHz) have recently aroused general interests [1], [2]. The mainstream 60GHz mm-Wave communication standards, i.e., IEEE 802.15.3c and 802.11ad [3], [4], are mainly oriented towards 5G communications aiming to support the high-speed data transmissions such as Uncompressed High Definition Television (UHDTV) and Bin Li, Chenglin Zhao, Mengwei Sun and Zheng Zhou are with the School of Information and Communication Engineering (SICE), Beijing University of Posts and Telecommunications (BUPT), Beijing, 100876 China. (Email: stonebupt@gmail.com). Haijun Zhang is with the Department of Information Engineering, Beijing University of Chemical Technology (BUCT), Beijing 100029, China. (Email: haijun.zhang@kcl.ac.uk). A. Nallanathan is with the Department of Informatics, King’s College Lon- don, London, WC2R2LS, United Kingdom. (Email: nallanathan@ieee.org). Manuscript received January 09, 2014. Gbps wireless accessing (e.g., Wi-Fi applications) [5], [6]. In all such scenarios, single carrier (SC) modulation has been recommended as one of the promising physical layer (PHY) techniques [7]. To achieve the ultra-high data rate up to 7Gbps with a regulated transmission bandwidth of 2.16GHz, high- order modulations such as M-order phase shift key (MPSK) and M-order quadrature amplitude modulation (M-QAM) have been commonly suggested to improve the spectrum efficiency furthermore [3], [4], [8]. As the absorption from oxygen to signals may reach a maximum in mm-Wave bands (about 15dB/km), in practice, the propagation attenuation in such a high-frequency band is tremendous, correspondingly coming with a rather limited link budget [9]. In order to efficiently compensate the path- loss and, therefore, reinforce the signal-to-noise ratio (SNR) in receivers, usually high emission power is used in 60GHz mm-Wave communications, except for the deployment of high resolution beam-forming techniques. Taking the high-order modulation signal with the high peak-to-average power ratio (PAPR) and the enormous bandwidth into accounts, mm- Wave communications may become extremely vulnerable to nonlinear distortions [10] and multipath propagations. Unfortunately, the mm-Wave power amplifier (PA) may inevitably show nonlinear characteristics due to hardware imperfections [11], which may arouse serious nonlinear dis- tortions in practice. The received signals, as a consequence, will be shifted sharply in the constellation-plane. Then, the symbols will be interpreted erroneously. In some bad cases, the bit error ratio (BER) will be significantly increased, leading to substantial performance deteriorations. In addition, further considering typical short-range indoor applications, the frequency-selective multipath propagations become very common to 60GHz mm-Wave systems. The resulting multi- path interference may also remarkably degrade the detection performance. Thus, one of the major concerns in mm-Wave communications is to combat the linear and nonlinear distor- tions and enhance the signal detection performance. In order to overcome the destructive effect, many investiga- tions have been dedicated to deal with the PA nonlinearity in mm-Wave systems. A simple and direct approach is to reduce the radiation power [12], which makes the operational power away from the PA’s saturation point and thereby alleviate nonlinear distortions to some extent. This output power back- off (OBO) method, nevertheless, may sacrifice the power efficiency and, as a consequence, the SNR in receiver may