Capacity Penalty due to Ideal Zero-Forcing Decision-Feedback Equalization John R. Barry, Edward A. Lee, and David G. Messerschmitt John R. Barry School of Electrical and Computer Engineering Georgia Institute of Technology Atlanta, Georgia, 30332-0250 barry@ee.gatech.edu (404) 894-1705 Edward A. Lee and David G. Messerschmitt Department of Electrical Engineering and Computer Sciences University of California Berkeley, California, 94720 {eal, messer}@eecs.berkeley.edu July 26, 1996 ABSTRACT We consider the capacity C of a continuous-time channel with frequency response H( f ) and additive white Gaussian noise. If |H( f )| –2 behaves like a polynomial of order ρ at high frequencies, we show that the per-symbol capacity approaches ρ ∕ 2 nats per channel use at high signal powers. If the receiver uses an ideal zero-forcing decision-feedback equalizer (DFE) consisting of a sampled whitened-matched filter followed by a zero-forcing tail canceller that is free of error propagation, the overall system is free of intersymbol interference and has a well-defined capacity C ZF . By comparing this capacity with the capac- ity C of the underlying channel, we quantify the loss of information inherent in the tail-cancelling opera- tion that typifies zero-forcing DFE and zero-forcing precoding systems. For strictly bandlimited channels, we find that the capacity penalty approaches zero in the limit of large signal power. On the other hand, for non-strictly bandlimited channels, the asymptotic penalty is nonzero; however, with bandwidth optimiza- tion, the asymptotic penalty is at most 0.59 dB, and the asymptotic ratio C ZF ∕C is at least 93.6%, depend- ing on the asymptotic order ρ of the channel response. Key words: Decision Feedback Equalization, Zero Forcing, Precoding. Presented in part at ICC ‘93, IEEE International Conference on Communications, Geneva, May 1993. Submitted to IEEE Transactions on Information Theory, January 1992; revised November 1995. As appeared in IEEE Transactions on Information Theory, vol. 42, no. 4, pp. 1062-1071, July 1996