WIRELESS COMMUNICATIONS AND MOBILE COMPUTING Wirel. Commun. Mob. Comput. 2014; 14:1564–1581 Published online 15 October 2012 in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/wcm.2301 RESEARCH ARTICLE Performance analysis of selective combining decode-and-forward relay networks over Nakagami-n and Nakagami-q fading channels Ehsan Soleimani-Nasab * , Mehrdad Ardebilipour and Ashkan Kalantari Faculty of Electrical and Computer Engineering, K. N. Toosi University of Technology,Shariati Street (before Seyyed Khandan Bridge), PO16315 1355, Tehran 1431714191, Iran ABSTRACT In this paper, we present the performance of selective combining decode-and-forward relay networks in independent and non-identically distributed Nakagami-n and Nakagami-q fading channels by using the best–worse and the decoding-set approaches. The outage probability, moment generation function, symbol error probability and average channel capacity are derived in closed-form using the signal to noise ratio (SNR) statistical characteristics. After that, we analyze the outage probability at high SNRs, and then, we optimize it. Beside the optimum method, we have proposed a sub-optimum adaptive method. Also, we derive the outage probability for the selection-combining case with the direct link between the source and the destination. Finally, for comparison with analytical formulas, we perform some Monte-Carlo simulations. Copyright © 2012 John Wiley & Sons, Ltd. KEYWORDS decode-and-forward; outage probability; symbol error probability; Nakagami-n fading; Nakagami-q fading; best–worse; decoding set; convex optimization *Correspondence Ehsan Soleimani-Nasab, Faculty of Electrical and Computer Engineering, K. N. Toosi University of Technology,Shariati Street (before Seyyed Khandan Bridge), PO16315 1355, Tehran 1431714191, Iran. E-mail: ehsan.soleimani@ee.kntu.ac.ir 1. INTRODUCTION Cooperative communication has been an interesting topic for researchers in recent years. Cooperative communica- tions refer to systems or techniques that allow users to help transmissions of each others to the destination. Most cooperative transmission schemes involve two phases of transmission: a coordination phase, where users exchange their own source data and control messages with each other and/or the destination, and a cooperation phase, where the users cooperatively retransmit their messages to the destination. To enable cooperation among users, different relay technology can be employed depending on the relative user location, channel condition, and transceiver complex- ity. Some of the basic cooperative relaying techniques are decode-and-forward (DF), amplify-and-forward (AF), coded cooperation, and compress-and-forward [1]. In this paper, we mainly focus on DF relay systems. DF relaying schemes refer to cases where the relay explic- itly decodes the transmitted message by the source and transmits a newly generated signal to the destination, as illustrated in Figure 1. There are two different DF relaying schemes. In basic DF, the relay is assigned to forward the source message in phase II, given that it has successfully decoded the message in phase I. However, in selection DF, the source is allowed to retransmit the message by itself in phase II [1]. Several selective combining schemes have been introduced in recent years. Wang and Giannakis in [2] investigated some general formulas for analyzing the effect of the fading channels in wireless networks. In [3], the authors introduced an opportunistic relaying method in which a single relay based on the best end-to-end instanta- neous signal to noise ratio (SNR) criterion is selected and then forwards the message to the destination. They derive analytical results at high SNRs. In [4], authors have calculated the outage and bit error probability for a dual-hop relay network for independent and identically distributed Nakagami-m fading channels with relays utilizing AF and DF protocols. In [5], the out- age probability of DF relaying system over Nakagami-m channels was presented. The authors in [6] and [7] 1564 Copyright © 2012 John Wiley & Sons, Ltd.