Achievable rates of SVD-based codebooks for zero-forcing and Tomlinson-Harashima precoding schemes with limited feedback MU-MIMO system Zeeshan Kaleem, Bing Hui and KyungHi Chang* ,† Electronic Engineering Department, Inha University, Incheon, Korea SUMMARY In this manuscript, we developed a new SVD-based codebook design criterion for linear zero-forcing (ZF) precoding and Tomlinson-Harashima (TH) precoding with limited feedback. In limited feedback systems where only quantized channel state information is available, precoding on transmitter causes sum-rate degradation because of quantization errors. Thus, we propose more efficient codebooks that reduce the quantization error with lower feedback overheads. We also develop a closed-form expression for sum-rate of the proposed codebooks in case of TH and ZF precoding schemes in a multiuser MIMO system with limited feedback. The numerical consequences prove that TH precoding can achieve sum-rate gain of 0.5–2.5 bps/Hz at 25 dB signal-to-noise ratio as compared with linear ZF precoding with 10 and 20 feedback bits per user, respectively. Moreover, the numerical results also prove that the performance of nonlinear TH precoding is better than linear ZF precoding for low signal-to-noise ratio value or for the large number of feedback bits. Copyright © 2016 John Wiley & Sons, Ltd. Received 16 August 2015; Revised 6 April 2016; Accepted 13 April 2016 KEY WORDS: SVD-based codebooks; ZF and TH precoding; MU-MIMO system; limited; feedback 1. INTRODUCTION Multiple-input multiple-output-based wireless systems have been proved spectrum efficient with high reliability [1]. Multiuser MIMO (MU-MIMO) wireless systems are beneficial because it can serve many users at once, which increase the system capacity as compared with single user system. MU-MIMO system performances decrease because of the MU interference, which occurs owing to the unavailability of full channel state information (CSI). For nonlinear Tomlinson-Harashima precoding (THP) has received much more attention compared with less complex zero-forcing (ZF) [2] and their improved version of channel regularization [3] for cancelling the MU interference at the base station (BS). The reason of gaining more attention by THP is its capability to reduce the sum-rate loss as compared with linear ZF precoding schemes. But to completely eliminate the interference, the THP matrix needs the full CSI on the BS side, which is unrealistic for frequency division duplexing systems without reciprocity. Because the quantized CSI based on a codebook is sent from each user to the BS over the feedback channel. Therefore, the accuracy of the CSI is necessary to completely cancel the interference during nonlinear processing on the BS side [4]. Thus, we need to increase the accuracy of channel quantization, that is, the accuracy of the designed codebooks with a reasonable feedback overhead. In literature, to reduce the quantization error by improving the quantization accuracy, the Grassmannian or line packing-based codebooks [5–7], predictive vector quantization (VQ)-based *Correspondence to: KyungHi Chang, Electronic Engineering Department, Inha University, Incheon, Korea. † E-mail: khchang@inha.ac.kr Copyright © 2016 John Wiley & Sons, Ltd. INTERNATIONAL JOURNAL OF COMMUNICATION SYSTEMS Int. J. Commun. Syst. (2016) Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/dac.3154