Digital Signal Processing 18 (2008) 307–320 www.elsevier.com/locate/dsp Performance analysis of a RLS-based MLP-DFE in time-invariant and time-varying channels Kashif Mahmood, Abdelmalek Zidouri, Azzedine Zerguine ∗ Electrical Engineering Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia Available online 27 April 2007 Abstract In this work, a recently derived recursive least-square (RLS) algorithm to train multi layer perceptron (MLP) is used in an MLP-based decision feedback equalizer (DFE) instead of the back propagation (BP) algorithm. Its performance is investigated and compared to those of MLP-DFE based on the BP algorithm and the simple DFE based on the least-mean square (LMS) algorithm. The results show improved performance obtained by the new structure in both time-invariant and time-varying channels. As will be detailed in this work, the newly proposed structure is a compromise between complexity and performance.  2007 Elsevier Inc. All rights reserved. Keywords: Multi layer perceptron (MLP); Decision feedback equalizer (DFE); Least-mean square (LMS); Recursive least-square (RLS) 1. Introduction A serious limitation in attempting to achieve a high transmission rate through a particular band-limited channel is the time dispersion suffered by the signal at the receiving end of this channel [1]. In data transmission, the time dispersion imparted on the transmitted signal results in a time overlap between successive symbols, known as inter- symbol interference (ISI). Equalizers have been used to describe filters used to compensate for such distortions in the amplitude and delay characteristics of the channel. Nonlinear equalizers [1,2] are superior to linear ones in applications where the channel distortion is too severe for a linear equalizer to handle. In particular, a linear equalizer does not perform well on channels with deep spectral nulls in their amplitude characteristics or with nonlinear distortion. A decision feedback equalizer (DFE) is a nonlinear equalizer that is widely used in situations where the ISI is very large. It has been proved theoretically and experimentally that the DFE performs significantly better than a linear equalizer of equivalent complexity [1]. The basic idea of DFE is that if the values of the symbols already detected are assumed to be correct, then the ISI contributed by these symbols can be canceled exactly by subtracting past symbol values with appropriate weighting from the equalizer output [2]. To further enhance the performance of the DFE, the multilayer perceptron (MLP) has been incorporated to the DFE. It is shown that the MLP-based DFE (MLP DFE) [3] and the MLP DFE with lattice structure [4], using the * Corresponding author. E-mail addresses: kashif_mahmood@hotmail.com (K. Mahmood), malek@kfupm.edu.sa (A. Zidouri), azzedine@kfupm.edu.sa (A. Zerguine). 1051-2004/$ – see front matter  2007 Elsevier Inc. All rights reserved. doi:10.1016/j.dsp.2007.04.006