Adaptive MIMO Reduced-Rank Equalization Based on Joint Iterative Least Squares Optimization of Estimators Rodrigo C. de Lamare †, Are Hjørungnes ‡ and Raimundo Sampaio-Neto § †Communication Research Group, Department of Electronics, The University of York, UK ‡UniK - University Graduate Center, University of Oslo, Norway §CETUC/Pontifical Catholic University of Rio de Janeiro (PUC-RIO), Brazil E-mails: rcdl500@ohm.york.ac.uk, arehj@unik.no, raimundo@cetuc.puc-rio.br Abstract— This paper presents a novel adaptive reduced-rank multi-input-multi-output (MIMO) linear equalization structure based on joint iterative optimization of adaptive filters. The proposed reduced-rank linear equalization structure consists of a joint iterative optimization of two equalization stages, namely, a projection matrix that performs dimensionality reduction and a reduced-rank linear equalization filter that retrieves the desired transmitted symbol. The novel linear reduced-rank structure is responsible for cancelling the inter- antenna interference caused by the associated data streams and exploiting the available degrees of freedom at the antenna-array receiver. We describe least squares (LS) expressions for the design of the projection matrix and the reduced-rank filter along with computationally efficient recursive least squares (RLS) adaptive estimation algorithms. Simulations for a MIMO linear equalization application show that the proposed scheme outperforms the state-of-the-art reduced-rank and the conventional estimation algorithms at about the same complexity. Index Terms— MIMO systems, linear equalization, parameter estimation, reduced-rank schemes. I. I NTRODUCTION T HE ever-increasing demand for performance and capac- ity in wireless communication networks has led to the investigation of many signal processing and communications techniques for employing the resources efficiently. Recent results on information theory have shown that it is possible to achieve high spectral efficiency [1] and to make wireless links more reliable [2], [3] via the deployment of multiple antennas at both transmitter and receiver. In MIMO communications systems, the received signal is composed by the sum of several transmitted signals which share the propagation environment and are subject to multiple propagation paths and noise at the receiver. The multipath channel gives rise to the intersymbol interference (ISI), whereas the non-orthogonality among the signals transmitted leads to multi-access interference (MAI) at the receiver. In order to mitigate the detrimental effects of ISI and MAI that reduce the performance and the capacity of MIMO This work was supported by the Research Council of Norway VERDIKT project 176773/S10 called OptiMO. systems, the designer has to construct a space-time or MIMO equalizer. The optimal MIMO equalizer known as the max- imum likelihood sequence estimation (MLSE) receiver was originally developed in the context of multiuser detection in [4]. However, the exponential complexity of the optimal MIMO equalizer makes its implementation costly for multi- path channels with severe ISI and MIMO systems with many antennas. In practice, designers often prefer the deployment of low-complexity MIMO receivers such as the linear [5], [6] and decision feedback equalizers (DFE) [7], [8]. The linear MIMO equalizer can offer a good trade-off between performance and complexity, and can be more easily analyzed than other non-linear structures. These receivers require the estimation of the coefficients used for combining the received data and extracting the desired transmitted symbols. A challenging problem which remains unsolved by conventional estimation techniques is that when the number of elements in the filter is large, the algorithm requires substantial training for the MIMO linear equalizer and a large number of received symbols to reach its steady-state behavior. Reduced-rank estimation [9]-[13] is a very promising and cost-effective technique in low-sample-support situations and in problems with long filters. The advantages of reduced-rank estimators are their faster convergence speed and better track- ing performance than existing techniques when dealing with large number of weights. Several reduced-rank methods and systems have been proposed in the last several years, namely, eigen-decomposition techniques [10], the multistage Wiener filter (MWF) [11], and the auxiliary vector filtering (AVF) algorithm [12]. Prior work on reduced-rank estimators for MIMO systems is extremely limited and relatively unexplored, being the work of Sun et al. [14] one of the few existing ones in the area. In this work, we propose a novel adaptive MIMO linear equalization structure based on a novel reduced-rank estima- tion method. The proposed reduced-rank equalization structure consists of a joint iterative optimization of two equalization stages, namely, a projection matrix that performs dimension- ality reduction and a reduced-rank estimator that retrieves the