IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 59, NO. 7, JULY 2011 3301 Globally Optimal Linear Precoders for Finite Alphabet Signals Over Complex Vector Gaussian Channels Chengshan Xiao, Fellow, IEEE, Yahong Rosa Zheng, Senior Member, IEEE, and Zhi Ding, Fellow, IEEE Abstract—We study the design optimization of linear precoders for maximizing the mutual information between finite alphabet input and the corresponding output over complex-valued vector channels. This mutual information is a nonlinear and non-concave function of the precoder parameters, posing a major obstacle to precoder design optimization. Our work presents three main con- tributions: First, we prove that the mutual information is a con- cave function of a matrix which itself is a quadratic function of the precoder matrix. Second, we propose a parameterized itera- tive algorithm for finding optimal linear precoders to achieve the global maximum of the mutual information. The proposed itera- tive algorithm is numerically robust, computationally efficient, and globally convergent. Third, we demonstrate that maximizing the mutual information between a discrete constellation input and the corresponding output of a vector channel not only provides the highest practically achievable rate but also serves as an excellent criterion for minimizing the coded bit error rate. Our numerical examples show that the proposed algorithm achieves mutual in- formation very close to the channel capacity for channel coding rate under 0.75, and also exhibits a large gain over existing linear precoding and/or power allocation algorithms. Moreover, our ex- amples show that certain existing methods are susceptible to being trapped at locally optimal precoders. Index Terms—Finite alphabet input, linear precoding, mutual information, optimization, vector Gaussian noise channel. I. INTRODUCTION L INEAR transmit precoding has been a popular research topic in multiple-input multiple-output (MIMO) system optimization, as evidenced by [1]–[17] and references therein. Existing methods typically belong to three categories: (a) di- versity-driven designs; (b) rate-driven designs; and (c) designs based on minimum mean squared error (MMSE) or maximum Manuscript received November 29, 2010; revised March 18, 2011; accepted March 18, 2011. Date of publication April 07, 2011; date of current version June 15, 2011. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Shahram Shahbazpanahi. The work of C. Xiao and Y. R. Zheng was supported in part by the NSF by Grants ECCS- 0846486 and CCF-0915846, and by the Office of Naval Research by Grants N00014-09-1-0011 and N00014-10-1-0174. The work of Z. Ding was supported in part by the NSF by Grants CCF-0830706 and ECCS-1028729. The material in this paper has been presented at the IEEE ICASSP 2011. C. Xiao and Y. R. Zheng are with the Department of Electrical and Computer Engineering, Missouri University of Science and Technology, Rolla, MO 65409 USA (e-mail: xiaoc@mst.edu; zhengyr@mst.edu). Z. Ding is with the Department of Electrical and Computer Engineering, Uni- versity of California, Davis, CA 95616 USA. He was on sabbatical leave at Southeast University, Nanjing, China (e-mail: zding@ucdavis.edu). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TSP.2011.2140112 signal-to-noise ratio (SNR). The first category applies pairwise error probability analysis to maximize diversity order as in [18] but may not achieve the highest coding gain [19]. The second category often utilizes (ergodic or outage) capacity as design criteria for precoder optimization. However, most such designs rely on the impractical Gaussian input assumption, which often leads to substantial performance degradation when applied with actual input of finite alphabet data [20], [21]. The third category uses MMSE or SNR as the figure of merit to design a linear pre- coder. Linear MMSE estimation strategy is globally optimal if the channel inputs and noise are both (independent) Gaussian [10]. However, when the inputs belong to finite alphabets, such strategy is also not optimal. In fact, several recent works have begun to study MIMO pre- coding design for maximizing mutual information under dis- crete-constellation inputs [20]–[24]. In [20], a seminal result was presented for power allocation based on mercury/water- filling (M/WF) that maximizes the input-output mutual infor- mation over parallel channels with finite alphabet inputs. The significance of M/WF is that, for given equal probable con- stellations, stronger channels may receive less allocated power; whereas for channels of equal strength, denser constellations may receive larger power allocation. These results indicate that power allocation depends not only on channel gain but also on the constellation for finite alphabet inputs. Therefore, M/WF is very different from the classic waterfilling (CWF) policy [25] for Gaussian inputs. However, M/WF inherits a feature of the CWF by constraining the M/WF precoding matrix as diag- onal if the channel matrix is also diagonal [20]. Unfortu- nately, this diagonal constraint on makes the M/WF power allocation sub-optimal even for parallel channels with discrete constellation inputs. The works in [21] and [22] proposed itera- tive algorithms based on necessary but not sufficient conditions. Hence, such algorithms do not guarantee global optimality. For real-valued vector channel models in which all signals and ma- trices [see (1)] have real-valued entries, [23] showed that: a) the mutual information between channel input and output, , is a function of with denoting matrix transpose; b) the left singular vectors of optimal can be the right singular vectors of ; c) the mutual information is a concave function of the squared singular values of if its right singular vectors are fixed. However, [23] pointed out that optimizing the right singular vectors of the precoder “seems to be an extremely diffi- cult problem”. Independently, [24] stated that is a con- cave function of for real-valued signals and chan- nels, with an incomplete proof. An iterative algorithm was fur- ther presented in [24] to solve the real-valued precoder . The 1053-587X/$26.00 © 2011 IEEE