IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 52, NO. 3, MAY 2003 525 Efficient Equalization and Symbol Detection for 8-PSK EDGE Cellular System Jan C. Olivier, Sang-Yick Leong, Chengshan Xiao, and Karl D. Mann Abstract—In this paper, a new method is presented for channel equalization and symbol detection of an enhanced data rate for global system for mobile (GSM) communication and IS-136 evo- lution (EDGE) cellular system in which eight phase-shift keying (8-PSK) modulation is employed. The new method iteratively min- imizes the Euclidean distance between the detected and received signal sequences, with neighbor symbol perturbation to reduce the computational complexity. The new algorithm is computationally efficient and can also be easily implemented into commercial signal processors. Simulation results comparing our method with the re- duced-state sequence-estimation (RSSE) method, both with and without set partitioning, are presented. Index Terms—Channel equalization, coherent symbol detection, enhanced data rate for global system for mobile (GSM) communi- cation and IS-136 evolution (EDGE) cellular system. I. INTRODUCTION T O meet the global demand for mobile communications and to be backward compatible with global system for mobile (GSM) communication and IS-136 time-division multiple-ac- cess (TDMA) mobile systems, European Telecommunications Standard Institute (ETSI) and Telecommunications Industry As- sociation (TIA) have adopted enhanced data rate for GSM and IS-136 evolution (EDGE) [1] as a third-generation standard. EDGE has a time frame and burst structure almost identical to that of GSM, but can achieve significantly higher data rates and spectral efficiency because it provides eight phase-shift keying (8-PSK) modulation. A simplified baseband system block dia- gram of EDGE with 8-PSK modulation is given in Fig. 1. The transmitted signal will be distorted by multipath fading and additive noise in the mobile radio channel. The channel equalizer is designed to jointly eliminate the intersymbol interference (ISI) and to estimate the transmitted symbol se- quence at the receiver. We may estimate a sequence rather than symbol-by-symbol estimation via the maximum likelihood Manuscript received May 31, 2001; revised September 4, 2002. This work was supported in part by the University of Missouri System Research Board, Columbus, MO, under Grant URB-02-124 and by Nortel Networks, Nepean, ON, Canada, is acknowledged. This paper was presented in part at the IEEE Vehicular Technology Conference (VTC’01 – Spring), Rhodes, Greece, May 2001. J. C. Olivier is with the Nokia Research Center, Irving, TX 75039 USA. S.-Y. Leong and C. Xiao are with the Department of Electrical and Com- puter Engineering, University of Missouri, Columbia, MO 65211 USA (e-mail: syleong@ee.missouri.edu; cxiao@ee.missouri.edu). K. D. Mann is with Wireless Solutions, Nortel Networks, Nepean, ON K2G 6J8, Canada. Digital Object Identifier 10.1109/TVT.2003.810991 sequence estimation (MLSE) [3]. A MLSE equalizer with a Viterbi algorithm is an optimal algorithm for minimizing the probability of sequence (or word) error, provided that the channel impulse response is known. The MLSE equalizer was widely postulated as a suitable technique for GSM radio with a binary Gaussian minimum-shift keying (GMSK) modulation scheme, but the introduction of 8 PSK will require the MLSE trellis to have 8 states, which is impossible to implement. To limit the computational complexity, some alternative methods were proposed in the literature. For example, the reduced-state sequence estimation (RSSE) with set partitioning algorithm [4] and the delayed-decision feedback-sequence estimation (DDFSE) [5] provide a good tradeoff between performance and computational complexity. Other reduced state trellis ap- proaches [based on maximum a posteriori probability (MAP) detection] are discussed in [6]. In this paper, a new approach is presented for 8-PSK EDGE channel equalization. The method is suboptimal but can achieve good bit-error ratio (BER) performance with low computational complexity. The algorithm iteratively minimizes the Euclidean distance between the detected and received signal sequences with good initial conditions provided in this paper. The detected hard symbols are then used to cancel ISI when computing the zero-delay form of the symbol (bit) probabilities for soft deci- sion decoding. The rest of the paper is organized as follows. Section II presents the channel equalization and symbol detection. Sec- tion III describes the simulation of the equalizer based on the ETSI channel models with 8-PSK modulation. Finally, Section IV draws the conclusions. II. CHANNEL EQUALIZATION AND SYMBOL DETECTION We assume burst-mode transmission in which the burst con- tains both data and pilot or training symbols as shown in Fig. 2. The training symbols enable channel estimation and the esti- mate is assumed valid over the entire burst. Based on a least-squares (LS) estimated overall-channel impulse-response vector , we can proceed to the channel equalization and information-symbol detection. With reference to Fig. 1 we indicate the need for a prefilter [8]. The prefilter transforms to a minimum-phase form in which the leading taps are dominant. Denoting the prefiltered received sequence by , we have (1) 0018-9545/03$17.00 © 2003 IEEE