1578 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 26, NO. 8, OCTOBER 2008 A Design Framework for Limited Feedback MIMO Systems with Zero-Forcing DFE Michael Botros Shenouda, Student Member, IEEE, and Timothy N. Davidson, Member, IEEE, Abstract—We consider the design of multiple-input multiple- output communication systems with a linear precoder at the transmitter, zero-forcing decision feedback equalization (ZF- DFE) at the receiver, and a low-rate feedback channel that enables communication from the receiver to the transmitter. The channel state information (CSI) available at the receiver is assumed to be perfect, and based on this information the receiver selects a suitable precoder from a codebook and feeds back the index of this precoder to the transmitter. Our approach to the design of the components of this limited feedback scheme is based on the development, herein, of a unified framework for the joint design of the precoder and the ZF-DFE under the assumption that perfect CSI is available at both the transmitter and the receiver. The framework is general and embraces a wide range of design criteria. This framework enables us to characterize the statistical distribution of the optimal precoder in a standard Rayleigh fading environment. Using this distribution, we show that codebooks constructed from Grassmann packings minimize an upper bound on an average distortion measure, and hence are natural candidates for the codebook in limited feedback systems. Our simulation studies show that the proposed limited feedback scheme can provide significantly better performance at a lower feedback rate than existing schemes in which the detection order is fed back to the transmitter. Index Terms—Limited feedback, Decision feedback equaliza- tion (DFE), Grassmann packings, Majorization, Schur-convexity. I. I NTRODUCTION M ULTIPLE-INPUT multiple-output (MIMO) communi- cation schemes offer the potential for significant in- creases in spectral efficiency over their single-input single- output counterparts by enabling simultaneous transmission of independent data streams. MIMO schemes also offer the potential for significant performance gains in a variety of other metrics. Standard transceiver architectures for these schemes include linear precoding and equalization, and the combination of linear precoding and decision feedback equalization (DFE), which offers the potential for improved performance over the linear approach while maintaining comparable complexity. For scenarios in which accurate channel state information (CSI) is available at both the transmitter and the receiver, there is a well established framework that unifies the design of linear transceivers under many design criteria [1]. A counterpart for the design of systems with DFE has recently emerged Manuscript received 4 November 2007; revised 15 April 2008. This work was supported in part by the Natural Sciences and Engineering Research Council of Canada and an Ontario Graduate Scholarship in Science and Technology. The work of the second author is also supported by the Canada Research Chairs Program. A preliminary version of this manuscript appears in Proc. Canadian Wkshp Inform. Theory, Edmonton, June 2007. The authors are with the Department of Electrical and Com- puter Engineering, McMaster University, Hamilton, Ontario, Canada ({botrosmw,davidson}@mcmaster.ca). Digital Object Identifier 10.1109/JSAC.2008.081023. [2]–[5]. This framework was also extended to MIMO sys- tems with pre-interference subtraction at the transmitter in [2]. However, in many scenarios, such as frequency division duplex systems, obtaining accurate CSI at the transmitter may require a considerable amount of feedback to the transmitter. An approach that allows the designer to limit the required amount of the feedback is to quantize the transmitter design. In these limited feedback schemes [6], the receiver uses its CSI to choose the best transmitter design from a codebook of available designs, and then feeds back the index of this precoder to the transmitter. This strategy has been considered for beamforming schemes (e.g., [7]–[13]), unitary precoding with linear equalization (e.g., [14]). and unitary precoding for orthogonal space time block codes [15], [16]. For zero-forcing DFE schemes, a limited feedback scheme in which the receiver feeds back the order of interference cancellation was proposed in [17], [18]. In this work, we consider the design of a limited feedback scheme for systems with a (general) linear precoder at the transmitter and zero-forcing DFE at the receiver. Our designs are based on a unified framework, developed herein, for the joint design of the precoder and the ZF-DFE in the presence of perfect CSI. This framework embraces a wide range of design criteria that can be expressed as functions of the mean square error (MSE) of each data stream, including minimization of the total MSE, minimization of the average bit error rate (BER), and maximization of the Gaussian mutual information. In particular, we show that the optimal precoder for systems with a zero-forcing DFE is the same for all these criteria; a property that cannot be achieved by a linear transceiver. Furthermore, we show that the optimal precoder for these objectives is a scaled unitary matrix that is isotropically distributed (over the Stiefel manifold of unitary matrices). Using this distribution, we show that codebooks constructed from Grassmann subspace packings minimize an upper bound on an average distortion measure, and hence are excellent candidates for the codebook in limited feedback schemes for systems with zero-forcing DFE. In contrast, the application of Grassmann codebooks in limited feedback schemes with linear receivers (e.g., [14]) involves an inherent compromise, because the optimal precoder in the presence of perfect CSI and a total power constraint is not unitary. Since the scheme that we propose involves the construction of codebooks for isotropi- cally distributed unitary matrices, our scheme subsumes that in [17], [18], in which the precoder is, by construction, a permutation matrix. Our simulation studies suggest that the additional degrees of freedom available in our approach enable our scheme to provide significantly better performance than that in [17], [18] while using a lower feedback rate. 0733-8716/08/$25.00 c  2008 IEEE Authorized licensed use limited to: McMaster University. Downloaded on July 13,2010 at 05:02:02 UTC from IEEE Xplore. Restrictions apply.