6298 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 12, DECEMBER 2010 Statistical Precoding With Decision Feedback Equalization Over a Correlated MIMO Channel Simon Järmyr, Student Member, IEEE, Björn Ottersten, Fellow, IEEE, and Eduard A. Jorswieck, Senior Member, IEEE Abstract—The decision feedback (DF) transceiver, combining linear precoding and DF equalization, can establish point-to-point communication over a wireless multiple-input multiple-output channel. Matching the DF-transceiver design parameters to the channel characteristics can improve system performance, but requires channel knowledge. We consider the fast-fading channel scenario, with a receiver capable of tracking the channel-state variations accurately, while the transmitter only has long-term, channel-distribution information. The receiver design problem given channel-state information is well studied in the literature. We focus on transmitter optimization, which amounts to designing a statistical precoder to assist the channel-tailored DF equalizer. We develop a design framework that encompasses a wide range of performance metrics. Common cost functions for precoder op- timization are analyzed, thereby identifying a structure of typical cost functions. Transmitter design is approached for typical cost functions in general, and we derive a precoder design formulation as a convex optimization problem. Two important subclasses of cost functions are considered in more detail. First, we explore a symmetry of DF transceivers with a uniform subchannel rate allocation, and derive a simplified convex optimization problem, which can be efficiently solved even as system dimensions grow. Second, we explore the tractability of a certain class of mean square error based cost functions, and solve the transmitter design problem with a simple algorithm that identifies the convex hull of a set of points in . The behavior of DF transceivers with optimal precoders is investigated by numerical means. Index Terms—Communication systems, decision feedback equalizers, fading channels, MIMO systems, precoders. I. INTRODUCTION I N recent years, multiple-input multiple-output (MIMO) wireless communication systems have received substantial attention. Promising a linear capacity increase in the number Manuscript received January 18, 2010; accepted August 05, 2010. Date of pub- lication August 30, 2010; date of current version November 17, 2010. The as- sociate editor coordinating the review of this manuscript and approving it for publication was Prof. Xiqi Gao. The research leading to these results has re- ceived funding from the European Research Council under the European Com- munity’s Seventh Framework Programme (FP7/2007-2013)/ERC Grant Agree- ment 228044. Part of the work of E. A. Jorswieck has been performed in the frame- work of the European research project SAPHYRE, which is partly funded by the European Union under its FP7 ICT Objective 1.1—The Network of the Future. S. Järmyr is with the ACCESS Linnaeus Center, Signal Processing Labo- ratory, Royal Institute of Technology (KTH), Stockholm SE-100 44, Sweden (e-mail: simon.jarmyr@ee.kth.se). B. Ottersten is with the ACCESS Linnaeus Center, Signal Processing Labo- ratory, Royal Institute of Technology (KTH), Stockholm SE-100 44, Sweden. He is also with the Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg (e-mail: bjorn.ottersten@ee.kth.se). E. A. Jorswieck is with the Dresden University of Technology, Communi- cations Laboratory, Communication Theory, Dresden 01062, Germany (e-mail: jorswieck@ifn.et.tu-dresden.de). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TSP.2010.2070499 of antennas under suitable propagation conditions [1], MIMO systems are well suited to combat the demand of reliable com- munication at increasingly higher data rates. In order to fully exploit the benefits of MIMO communications, the transceiver architecture needs to be carefully selected and optimized. Interestingly, when embedded in a suitable BLAST (Bell Lab- oratories Layered Space-Time) architecture—either vertical [2] or diagonal [3]—decision feedback (DF) equalization has the potential to achieve the information-theoretic performance bounds for both fading and static MIMO channels [4], [5]. Channel coding can be utilized to ensure reliable communica- tion in certain transmission scenarios; the probability of detection error can be made arbitrarily small by using codewords of ever-in- creasing length. In communication systems with a delay con- straint,uncodedtransmissionisaviablealternative.Inthissetting, the maximum likelihood (ML) detector is known for its excellent performance,butatthecostofhighcomputationaldemands.Inde- pendent subchannel detection assisted by linear equalization, on the other hand, is highly efficient but compromises performance [6]. With DF equalization, enhanced performance is achieved at only a marginal complexity increase [7, Ch. 4]. In most practical scenarios the wireless MIMO channel is far from static; the channel is sensitive to antenna movements on both sides, as well as to movements of other objects in the sur- roundings. In terms of system performance, it is material that the receiver has access to channel-state information (CSI) [8]. Channel estimation can be performed on the receiver side if, for example, a suitable pilot sequence is transmitted. With a slow-fading channel, it is possible to convey a quantized esti- mate to the transmitter on a reverse link. In a fast-fading envi- ronment, it is reasonable to assume that the transmitter is only aware of channel-distribution information (CDI). The detection at the receiver can be facilitated by a cleverly designed transmitter. With a CSI-aware transmitter and optimal linear precoding, DF equalization has a noticeable performance gain over the linear equalizer for a wide range of design criteria [7]. However, it was recently realized that this gain vanishes when rate-optimized systems are considered [9], since the op- timal DF design reduces to a linear transceiver [10]. The statis- tical, CDI-based precoder design problem was addressed in [11] for the linear equalizer, and in [12] and [13] for the DF equal- izer. While [13] minimizes the joint error probability (JEP) of the transmitted symbols assuming an uncorrelated channel, the design in [12] targets expected mean square error (MSE) and a channel model allowing spatial correlation. The present paper considers optimal statistical precoding with independent subchannel detection and DF equalization, over a channel with arbitrary correlation at the transmitter. The receiver is assumed to have access to CSI, while trans- mitter optimization relies on channel statistics via CDI. We 1053-587X/$26.00 © 2010 IEEE