Wireless Personal Communications 20: 41–60, 2002. © 2002 Kluwer Academic Publishers. Printed in the Netherlands. Diversity Combining and Equalization with TCM for Wireless Communications YUE CHEN 1 , K.B. LETAIEF 2 ⋆ and JUSTIN C.-I. CHUANG 3 1 Mobilink Telecom, Inc., 2710 Walsh Avenue, No. 200, Santa Clara, CA 95051, U.S.A. 2 Electrical and Electronic Engineering Department, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, P.R.China Corresponding author: E-mail: eekhaled@ee.ust.hk 3 AT&T Labs – Research, Holmdel, NJ 07733-0400, U.S.A. Abstract. This paper presents an equalization structure in which antenna diversity, adaptive decision feedback equalization (DFE), interleaving and trellis-coded modulation (TCM) can be effectively combined to combat both ISI and cochannel interference in cellular mobile radio environments. The feedback filter of the DFE can use either tentative or final decision symbols of the TCM Viterbi decoding to cancel tail ISI with the square root Kalman algorithm. A performance bound on the average pairwise error probability for TCM under perfect interleaving and equalization is obtained by analysis. Some simulation results which illustrate the potential of the proposed system will also be given. In particular, a performance comparison between the proposed method and uncoded QPSK modulation will be undertaken. Keywords: trellis-coded modulation, adaptive equalization, interleaving, diversity, frequency-selective fading channel, cochannel interference. 1. Introduction In a mobile digital communications environment, multipath propagation causes dispersion of the transmitted signal. The spread of time delay degrades system performance by causing intersymbol interference (ISI). This impairment has become an important issue [1, 2] for high- speed signaling rates and broadband wireless access. In [3], we investigated the performance of unequalized trellis-coded modulation (TCM) in flat and frequency-selective fading chan- nels. The simulation results show that TCM schemes with perfect interleaving outperforms uncoded QPSK at low delay spread. However, at relatively high delay spread (>0.2 symbol period), TCM schemes do not have advantages over QPSK and they both perform poorly without equalization. In previous work, the effects of ISI on the performance of equalized TCM in a frequency- selective fading channel has been investigated [4–10]. The performance of TCM and QPSK with maximum likelihood sequence estimation (MLSE) and D-branch diversity has been also considered in [4, 5]. These papers show that TCM does not outperform QPSK. This may be due to the fact that interleaving was not considered in the TCM scheme. Generally, MLSE is an optimum detection technique, however, joint MLSE, deinterleaving and TCM decoding is difficult to perform. Moreover, this requires knowledge of the channel impulse response and the amount of computation required increases exponentially with the length of the channel taps. In this paper, we consider a linear equalizer (LE) and decision-feedback equalizer (DFE)