International Journal of Computer Applications (0975 8887) Volume 88 - No. 1, February 2014 Error Analysis of 2-tier M -ary Star QAM Modulation in Shadowed Fading Channels Sourjya Dutta Polaris Networks Salt Lake Electronics Complex Kolkata 700091, India Iti Saha Misra Deptt. of Electronics and Telecommunication Engg. Jadavpur University Kolkata 700032, India ABSTRACT The error performance of the 2-tier star shaped Quadrature Amplitude Modulation scheme over K and K G fading channels are analyzed and evaluated. Novel closed form expressions for Symbol Error Rate (SER) have been derived for M -ary 2-tier circular Star QAM transmitted over the K and K G fading channels. The expressions derived are in the form of sum of single definite integrals of hypergeometric functions which are calculated using numerical methods. The expressions are validated by extensive Monte Carlo simulation. A simple relationship between SER and bit error rate (BER) is proposed and experimentally verified. Using the expressions for SER, the optimum values of ring ratio are calculated for various values of M . It has also been established that the error performance of 2-tier Star QAM is considerably superior to that of the M -ary Square QAM for high and moderate fading. General Terms: Digital Modulation, Wireless Channel Keywords: Star Quadrature Amplitude Modulation (QAM), Rayleigh-Gamma(K), Nakagami-Gamma(K G ), Fading, Shadowing, Symbol Error Rate(SER), Bit Error Rate(BER) 1. INTRODUCTION Various statistical models are used to study the effects of multipath fading and shadowing that severely degrade the performance of wireless communication systems. Effects of small-scale multipath fading on communication systems have been widely studied. The analysis of the effect of shadowing in wireless communication systems had been impeded by the complex mathematical form of the log-normal distribution conventionally used to quantify this phenomenon. In relatively recent works the K and the K G fading models have been found to be a suitable replacement of the log-normal distribution to estimate the combined effects of shadowing and small-scale fading in wireless media [1, 2, 3]. Moreover the Generalized-K distribution is also being used to model the effect of turbulence in free space optical (FSO) communication channels [4, 5]. The mathematical forms of these distributions allow closed form integration and hence system parameters can be efficiently computed using numerical techniques. Error analysis of lower order M-ary Phase Shift Keying (PSK) and square M-ary Quadrature Amplitude Modulation (MQAM) schemes for K and K G channels is found in [3, 6, 7]. Similar analysis for higher order and complex modulation schemes are absent in literature. M-ary QAM schemes are spectrally efficient modulation schemes which ensures higher data rates without requiring extra bandwidth. It has been pointed out in [8] that the widely used square MQAM scheme will have a high chance of false phase locking in channels where both amplitude and phase of the transmitted signal may vary considerably. The authors in [8] have contended that the M-ary Star QAM modulation scheme will be better suited for vehicular environments. In [9] it has been shown that the M-ary Star QAM scheme gives a better performance in heavy fading channels like the Rayleigh channel and also simplifies the receiver structure as automatic gain control (AGC) and carrier recovery are no longer required. As given in [10], the M-ary Star QAM overcomes the problem of high peak-to-average power ratio (PAPR) present in square and rectangular QAM schemes. In recent academic literature [11, 12, 13, 14] the Star QAM modulation has been studied for applications in optical communication. Additionally in [15] it has been pointed out that circular QAM schemes provide better performance than rectangular QAM for quantum detection. Error analysis of Star MQAM schemes in small-scale multipath fading channels can be found in [16, 17, 18] but to the best of our knowledge error rate estimation of Star QAM in shadowed fading channels is absent in current literature. The aim of this paper is to analyse the combined effects of shadowing and multi-path fading on the performance of 2-tier M-ary Star QAM schemes. The contributions of this paper are : i. Derivation of numerically computable expressions for average SER of Star MQAM considering the effects of both multipath fading and shadowing. ii. Derivation of an efficient relationship between average SER and average BER for the 2-tier Star QAM. iii. Proposing optimum values of ring ratio for different constellation sizes in composite multi-path and shadow fading channel. iv. To compare the error performance of the Star and the Square constellations in the discussed propagation model. The rest of the paper has been structured as follows. In Section 2 the channel models used are discussed. The theoretical derivation 9