Performance Analysis of Mixed FSO/RF Relay Systems in Shadowed LOS/NLOS Fading Imène Trigui, Nesrine Cherif, and Sofiène Affes INRS-EMT, 800, de la Gauchetière Ouest, Bureau 6900, Montréal, H5A 1K6, Qc, Canada. {itrigui, nesrine.cherif, affes}@emt.inrs.ca Abstract—This letter presents a unified analytical framework for the computation of the ergodic capacity and the outage probability of relay-assisted mixed FSO/RF transmission. In addition to accounting for different FSO detection techniques, the mathematical model offers a twofold unification of mixed FSO/RF systems by considering mixed Málaga-M/κ-μ shadowed fading, which includes as special cases nearly all linear turbulence/fading models adopted in the open literature. Index Terms—Amplify-and-forward (AF), ergodic capacity, free-space optics (FSO), κ-μ shadowed fading, Málaga-M dis- tribution, outage probability, pointing errors. I. I NTRODUCTION Recently, free-space optical (FSO) communications have gained a significant attention due to their advantages of higher bandwidth in unlicensed spectrum and higher throughput com- pared to their RF counterparts [1]. Hence, the gathering of both FSO and RF technologies arises as a promising solution for se- curing connectivity between the RF access and the fiber-optic- based backbone networks. As such, there has been prominent interest in mixed FSO/RF systems where RF transmission is used at one hop and FSO transmission at the other [2]-[5]. Most contributions within this research line consider restrictive irradiance and channel gain probability density function (PDF) models for the FSO and RF links, respectively. The most commonly utilized models for the irradiance in FSO links are the lognormal and the Gamma-Gamma (G-G) ([4],[5] and ref- erences therein). Recently, a new generalized statistical model, the Málaga-M distribution, was proposed in [6] to model the irradiance fluctuation of an unbounded optical wavefront propagating through a turbulent medium under all irradiance conditions. Characterized in [7] as a mixture of Generalized- K and discrete Binomial distributions, the Malága-M distri- bution unifies most statistical models exploited so far and is able to better reflect a wider range of turbulence conditions [6], [7]. On the RF side, previous works typically assume either Rayleigh or Nakagami-m fading [4], [5], thereby lacking the flexibility to account for disparate signal propagation mecha- nisms as those characterized in 5G communications which will accommodate a wide range of usage scenarios with diverse link requirements. To bridge this gap in the literature, the κ-µ Work supported by the Discovery Grants and the CREATE PERSWADE programs of NSERC, a Discovery Accelerator Supplement (DAS) Award from NSERC, and the NSERC SPG Project on Advanced Signal Processing and Networking Techniques for Cost-Effective Ultra-Dense 5G Networks. This paper is the first conference version of a published letter [2] in IEEE WCL. shadowed fading model, recently derived in [8], is an attractive proposition. In addition to offering an excellent fit to the fading observed in a range of real-world applications (e.g. device- to-device, and body-centric fading channels [9]), the κ-µ shadowed fading encompasses several RF channel models such as Nakagami-m, Rayleigh, Rice, κ-µ and shadowed Rician fading distributions. This new channel fading model offers far better and much more flexible representations of practical fading LOS (line of sight), NLOS (non-LOS), and shadowed channels than the Rayleigh and Nakagami-m distributions. Under the assumption of AF relaying and taking into account the effect of pointing errors while considering both heterodyne and intensity modulation/direct (IM/DD) detection techniques, we derive closed-form expressions for the ergodic capacity and outage probability of dual-hop FSO/RF systems over Málaga-M/κ-µ shadowed channels. To the best of the authors’ knowledge, for mixed FSO/RF AF systems, only [10] has considered the Málaga-M for the FSO link, let alone the κ-µ shadowed case. II. CHANNEL AND SYSTEM MODELS We consider a relay-assisted mixed FSO/RF transmission composed of both Málaga-M with pointing errors and κ- µ shadowed fading environments. The source communicates with the destination through an intermediate relay, able to activate both heterodyne and IM/DD detection techniques at the reception of the optical beam. The FSO (S-R) link irradiance is assumed to follow a Málaga-M distribution with pointing errors impairments for which the PDF of the irradiance, I , is given by [6, Eq. (5)] f I (x)= ξ 2 A xΓ(α) β  k=1 b k Γ(k) G 3,0 1,3  αβ gβ +Ω x A 0      ξ 2 +1 ξ 2 , α, k  , (1) where ξ is the ratio between the equivalent beam radius and the pointing error displacement standard deviation (i.e., jitter) at the relay (for negligible pointing errors ξ →∞), A 0 defines the pointing loss [1], A = α α 2 [gβ/(gβ + Ω)] β+ α 2 g −1− α 2 and b k = ( β − 1 k − 1 ) (gβ + Ω) 1− k 2 [(gβ +Ω)/αβ] α+k 2 (Ω/g ) k−1 (α/β ) k 2 , where α, β, g, and Ω are the fading parameters related to the atmospheric turbulence conditions [6]. Moreover in (1), G m,n p,q [·] and Γ(·) stand for the Meijer-G [11, Eq. (9.301)] and the incomplete gamma [11, Eq. (8.310.1)] functions, respectively. It is worth highlighting that the M distribution unifies most of the proposed statistical models characterizing the optical irradiance in homogeneous and isotropic turbulence