Multicell LMMSE Filtering Capacity under Correlated Multiple BS Antennas (Invited Paper) Symeon Chatzinotas ∗ , Muhammad Ali Imran † , Reza Hoshyar † , Bj¨ orn Ottersten ∗‡ ∗ SnT - securityandtrust.lu, University of Luxembourg, Email: {Symeon.Chatzinotas, Bjorn.Ottersten}@uni.lu † Centre for Communication Systems Research, University of Surrey, UK, Email: {M.Imran, R.Hoshyar}@surrey.ac.uk ‡ Royal Institute of Technology (KTH), Sweden, Email: bjorn.ottersten@ee.kth.se Abstract—Multicell joint processing has been shown to effi- ciently suppress inter-cell interference, while providing a high capacity gain due to spatial multiplexing across distributed Base Stations (BSs). However, the complexity of the optimal joint decoder in the multicell uplink channel grows exponentially with the number of users, making it prohibitive to implement in prac- tice. In this direction, this paper investigates the uplink capacity performance of multicell joint linear minimum mean square error (LMMSE) filtering, followed by single-user decoding. The considered cellular multiple-access channel model assumes both Rayleigh and Rician flat fading, path loss, distributed users and correlated multiple antennas at the base station side. The case of Rayleigh fading is tackled using a free probability approach, while the case of Rician fading is addressed through a de- terministic equivalent calculated using non-linear programming techniques. In this context, it is shown that LMMSE can provide high spectral efficiencies in practical macrocellular scenarios. I. I NTRODUCTION Since Wyner [1] introduced the concept of Base Station (BS) cooperation in the research community, the performance of optimal multicell decoding in the uplink channel has been extensively investigated, showing a high potential for capacity enhancement. According to this paradigm, the BSs are interconnected through reliable links (backhaul) to a central processor, which is assumed to have perfect channel state information (CSI) and strong processing capabilities, allowing for joint decoding of all system users. During the last decade, multicell decoding models have gradually evolved by incorporating more realistic characteristics of the wireless cellular channel, such as fading and path loss [2], [3], [4], user distribution [5], clustering [6], multiple antennas [7], [8] and fading correlation [9]. In order to achieve the optimal capacity in a cellular multiple-access channel, all the User Terminals (UTs) have to transmit simultaneously over the ensemble of the channel time-frequency resources, while Successive Interference Can- cellation (SIC) is utilized at the joint processor [10], [2], [11] to recover the individual user signals. However, the complexity of such a receiver grows exponentially with the number of UTs [12] and additionally, successive techniques can introduce error propagation in the decoding process. What is more, the backhaul network needs to be able to accommodate all the data traffic between the BSs and the central processor. In this direction, this paper investigates the capacity per- formance of the reduced-complexity Linear Minimum Mean Square Error (LMMSE) receiver, which can still exploit the paradigm of BS cooperation. More specifically, the computationally-expensive multicell joint decoder is replaced by an LMMSE filter [13], [14], which aims at jointly maximiz- ing the achieved Signal to Interference and Noise Ratio (SINR) across all the cooperating cells. The outputs of the filter are subsequently fed into conventional single-user decoders. At this point, it should be noted that LMMSE filtering can also be performed in a distributed fashion, alleviating the need of carrying the received analog observtion vectors of all BSs to the central processor [15], [16]. The main limitation of the LMMSE receiver is that the number of users that can be effectively filtered is limited by the rank of the channel matrix, namely the total number of BS antennas in the system. In case the number of UTs exceeds the number of BS antennas, it can be decreased by splitting the intra-cell UTs into orthogonal groups using TDMA or FDMA techniques [2], [4], [3]. A relevant investigation of LMMSE filtering capacity can be found in [4], although therein a single multiple-antenna UT per cell is considered in combination with single-cell linear MMSE detectors or nonlinear MMSE SIC detectors. In our investigation, multiple single-antenna UTs per cell can transmit simultaneously, as long as the total number of UTs per cell matches the number of receive antennas at the BS. In addi- tion, the authors in [17] investigate the performance of global LMMSE receiver for the Wyner’s circular system, assuming collocated users and non-faded non-correlated channels for the purposes of analysis. Moreover, in [17] the LMMSE receiver is applied across groups of intra-cell users, while optimal joint decoding is utilized to recover the individual user signals. In our case, the LMMSE filter is applied across all system users, followed by single-user decoding. Furthermore, we introduce a comprehensive cellular multiple-access channel model, which considers both Rayleigh and Rician flat fading, path loss, distributed UTs and correlated multiple antennas at the BS- side. The remainder of this paper is organised as follows. In the next section, we define the cellular multiple-access channel model for both Rayleigh and Rician Fading. In section III we describe the derivation of capacity expressions for the LMMSE receiver. In section IV, we evaluate and compare the capacity results produced by both simulation and analysis in the context of a typical macrocellular scenario. The last section concludes 978-1-4244-3574-6/10/$25.00 ©2010 IEEE