3002 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 10, NO. 9, SEPTEMBER 2011 Unified Performance Analysis of Two-Hop Amplify-and-Forward Relay Systems with Antenna Correlation Nuwan S. Ferdinand, Student Member, IEEE, and Nandana Rajatheva, Senior Member, IEEE Abstract—We present a unified performance analysis of a system in which the source and the destination are equipped with multiple antennas and communicating via a single antenna relay. Our studies can be divided into two parts. First we consider a system with maximal ratio transmission (MRT) at the source and maximal ratio combining (MRC) at the destination by assuming general correlation structures with arbitrary eigenvalue multi- plicities at the source and the destination. Then a system with transmit antenna selection (TAS) at the source with uncorrelated antennas and MRC at the destination with correlated antennas is investigated. The exact closed form expressions for outage probability, average symbol error rate (SER), generalized higher moments of SNR for both channel state information (CSI)- assisted and fixed gain relaying are derived and an analysis of the ergodic capacity is provided. Hence, our new results cover several previously reported cases as well as new additional ones. Further, we present the asymptotic analysis which gives an insight of the system performance and the diversity gain in each case. To verify the analytical results we provide Monte Carlo simulations at the end. Index Terms—CSI-assisted relaying, fixed gain relaying, MRT, TAS, antenna correlation. I. I NTRODUCTION B OTH CSI-assisted and fixed gain two-hop amplify-and- forward (AF) relays with a source having multiple an- tennas communicating with multiple antenna destination via a single antenna relay have been investigated in numerous previous literature. The analysis [1]–[9] considered MRT and MRC to improve the system performance. A. Related Previous Work Authors in [2] presented the performance of two-hop CSI- assisted relay network over independent Rayleigh fading chan- nels with MRT at the source and MRC at the destination. The performance of the same system, considering the antenna correlation effects at the source and the destination has been Manuscript received October 10, 2010; revised April 6, 2011; accepted May 26, 2011. The associate editor coordinating the review of this paper and approving it for publication was S. Sfar. This work has been funded in part by the Academy of Finland Grant # 128010, UNICS and the LOCON Project. N. S. Ferdinand is with the field of ICT, School of Engineering and Tech- nology, Asian Institute of Technology P.O. Box 4, Klong Luang, Pathumthani 12120, Thailand (e-mail: nuwanferdinand@gmail.com). N. Rajatheva is an Associate professor with Asian Institute of Technology, Klong Luang, Pathumthani 12120, Thailand, and a visiting Professor at the Centre for Wireless Communications, University of Oulu, Finland (e-mail: rajath@ait.ac.th, rrajathe@ee.oulu.fi). Digital Object Identifier 10.1109/TWC.2011.072511.101783 analyzed in [3]. Recently, in [6] and [7], the performance of a dual-hop fixed gain network over independent Rayleigh and Nakagami- fading environments has been analyzed. The fixed gain relaying with exponential correlation is reported in [8]. Authors in [10] introduced a different system model of a single antenna source communicating with a destination equipped with multiple antennas through a CSI-assisted relay having multiple receive antennas employing selection com- bining (SC) and a single transmit antenna. They have derived the closed form solutions for two cases; dual correlated and multiple uncorrelated antennas at the relay, with no closed form solution for multiple correlated antennas case. For the above cases they have considered correlated antennas at the destination, where correlation matrices have distinct eigenval- ues. B. Contributions Use of multiple antennas increases the system performance, where as it decreases with the antenna correlation. It is important to investigate this performance loss due to antenna correlation. To the best of our knowledge, the previous studies have only considered specific cases of correlation in CSI- assisted and/or fixed gain relaying. This paper aims to provide a unified analysis of MRT/MRC for a two-hop AF relay system with arbitrary correlation at the source and the destination. Further, we investigate the performance of the TAS/MRC for the same system with uncorrelated antennas at the source and correlated antennas at the destination. We extend our analysis and results in [9] which was limited to fixed gain to CSI-assisted relaying. A general fading model with arbitrary eigenvalue distribution for the cor- relation matrices is assumed in which eigenvalues need not be distinct, which allows us to investigate new correlation matrix structures [e.g. Uniform correlation [11, Eq.6.b]]. Specifically, we derive the exact closed form solutions to outage probability, average SER, generalized higher moments of SNR and provide ergodic capacity analysis for both CSI-assisted and fixed gain relay schemes. In order to gain insights, we derive asymptotic expressions in high SNR. II. SYSTEM MODEL Consider a communication system where a source (S) equipped with  T antennas, communicates with a destination (D), equipped with  R antennas, via a single antenna relay (R). 1536-1276/11$25.00 c ⃝ 2011 IEEE