Improving the Spectral Efficiency of Amplify-and-Forward Cooperative Relay Network with Adaptive M-QAM Modulation Bhuvan Modi and A. Annamalai Center of Excellence for Communication Systems Technology Research Department of Electrical & Computer Engineering Prairie View A&M University, TX 77446, USA ABSTRACT — This paper investigates the performance of cooperative amplify-and-forward (CAF) relay networks with adaptive M-ary quadrature amplitude modulation (M-QAM) technique in Nakagami-m and Rice wireless fading environments. We derived closed form upper bound expression for the marginal MGF of end-to-end SNR of CAF relay system in Nakagami-m fading to compute average bit error rate (ABER), mean achievable spectral efficiency, and outage probability performance metrics. Additionally, along with the above bounding techniques, we have adopted an accurate approximate MGF of end-to-end SNR of CAF relay system to compute above mentioned performance metrics. Employing the above novel approach based on “approximate MGF” allows us to simplify the computation complexity of achievable spectral efficiency as well as ABER of CAF relay system in generalized fading environments by simply replacing appropriate single channel MGF as readily available. The accuracies of our analytical results have been validated via Monte Carlo simulations. Index Terms ---- - cooperative communications, wireless link adaptation, adaptive M-QAM modulation. I. MOTIVATION The demand for wireless communication system in recent years has necessitated a need to improve the performance, reliability and data rate of wireless channels. To address these needs, several attempts have been made on each of these improvement areas. The broadcast nature of wireless transmissions has enabled a new communication paradigm known as “cooperative communication” where the source node communicates with a destination node with the help of one or more relay nodes to harness the new form of spatial diversity and combat multipath fading. This introduced additional reliability improvements. Adaptive transmission is yet another powerful wireless communication technique for improving the spectral efficiency wherein the signal constellation size, power level and/or coding rate are “matched” to the prevailing channel conditions based on the acquired channel-side-information (CSI) on the feedback channel. Although there have been extensive prior work on performance analyses of non-adaptive (i.e., fixed-rate or fixed-power) cooperative diversity systems and also on adaptive transmission techniques for traditional (non- cooperative) wireless networks, the art of adaptive link layer in cooperative wireless networks is still in its infancy [1]-[3]. But theoretical studies of such systems would be of interest to several emerging IEEE 802 standards such as 802.11s and 802.16j that are considering the use of cooperative techniques in their multi-hop relay architecture for enhancing coverage, throughput, and network capacity. In this paragraph a brief review of prior research on cooperative diversity with adaptive transmission is discussed. The problem of optimal power allocation in wireless relay networks was investigated in [4]-[5], but source rate- adaptation was not considered. In [6], the author investigated the adaptive transmission for a two-hop (regenerative and non- regenerative) relay network (but without the direct source- destination link) in Rayleigh fading. [7], [8] derived bounds for the Shannon capacity of link adaptive CAF relay networks with limited CSI, in which the rate and/or power level at the source node is adapted according to changing channel condition. The performance of CAF with constant power M-QAM adaptive rate transmission when the ABER in Rayleigh fading is constrained to be below a specified target bit error rate (BER) is examined in [2] and [3] for fixed and optimum mode switching thresholds, respectively. In [9], the performance of discrete-rate adaptive M-QAM for a single incremental relay in Nakagami-m environment was examined. In [10], we presented a new analytical framework based on marginal MGF in conjunction with Fixed-Talbot method [11] for the computation of ABER, average spectral efficiency and outage probability performance metrics, which allows us to convert MGF to CDF. However, the results are only presented for Nakagami-m fading environments. In this work, we presented the result for Rice fading channel using the above technique. However, motivated by the need for closed form solution, derived closed form expression for the marginal MGF of end-to-end SNR of CAF relay system by the use of partial fraction and employ the resulting expression in direct evaluation of performance metrics of CAF relay system. However, it is worth to mention that, the partial expansion decomposition is only applicable for integer m and more tedious for multiple relay system. Therefore, from computational efficiency perspective, it is considerably simpler to program and evaluate the desired ABER, mean spectral utilization efficiency and outage probability, using closed form MGF (in conjunction with Fixed-Talbot method [12]) for non-integer m and high diversity order for any fading environments (i.e., since the MGF of total received SNR may This work is supported in part by funding from the US Army Research Office (Cooperative Agreement W911NF-04-2-0054), US Air Force Research Laboratory, and the National Science Foundation (0931679 & 1040207). 978-1-4577-0638-7 /11/$26.00 ©2011 IEEE