Outage Performance for Amplify-and-Forward Channels with an Unauthenticated Relay Jing Huang, Amitav Mukherjee, and A. Lee Swindlehurst Electrical Engineering and Computer Science University of California, Irvine, CA 92697 Email: {jing.huang; amukherj; swindle}@uci.edu Abstract—We investigate a relay network where the source can potentially utilize an unauthenticated amplify-and-forward (AF) relay to augment its direct transmission of a confidential message to the destination. Since the relay is unauthenticated, it is desirable to protect the confidential data from it while simultaneously making use of it to increase the reliability of the transmission. We study the likelihood of achieving simultaneously secure and reliable message transmission via the secrecy outage probability (SOP) of the relay network. We first characterize the SOP for three different schemes: direct transmission, conven- tional AF relaying, and cooperative jamming. Subsequently, an asymptotic analysis is conducted to determine the optimal policies for different power budgets and channel gains. Numerical results are presented to verify the theoretical predictions of the preferred transmission policies from a secrecy outage perspective. I. I NTRODUCTION There has recently been intensive interest in improving information security at the physical layer of wireless networks [1], [2]. Besides the widely-used metric of secrecy rate, an alternative secrecy criterion that has recently been investigated for fading channels is the secrecy outage probability, which describes the probability of simultaneously reliable and secure data transmission [3]. In the context of relay channels, the secrecy outage probabil- ity has been investigated in [4], [5] for networks compromised of external eavesdroppers that are distinct from the source/sink and relay nodes. However, even if external eavesdroppers are absent, it may be desirable to keep the source signal confidential from the relay node itself in spite of its assistance in forwarding the data to the destination [6]. For example, the unauthenticated relay may belong to a heterogeneous network without the same security clearance as the source and destina- tion nodes. This scenario has also been denoted as cooperative communication via an untrusted relay in [7], where the authors presented bounds on the achievable secrecy rate. Furthermore, they showed that amplify-and-forward (AF) and compress- and-forward relaying (including a direct link) admit a non- zero secrecy rate even when the relay is untrusted, which does not hold for decode-and-forward relaying. Therefore, in this work we focus on the AF relaying protocol due to its increased security vis-` a-vis decode-and-forward, and its lower complexity as compared to compress-and-forward. This work was supported by the U.S. Army Research Office under the Multi-University Research Initiative (MURI) grant W911NF-07-1-0318. This paper analyzes a three-node network where the source can potentially utilize an unauthenticated relay to augment the direct link to its destination. The relay thus is in effect also an eavesdropper, even though it complies with the source’s request to act as a relay to the destination. We characterize the exact secrecy outage probability of three different transmis- sion policies: direct transmission where the relay is ignored, conventional AF relaying, and cooperative jamming by the destination to selectively degrade the relay’s eavesdropping capability. Subsequently, an asymptotic analysis of the outage probabilities is conducted to elicit the optimal policies for dif- ferent power budgets and channel gains. Among our findings, we demonstrate that direct transmission is best when either of the relay hops is weak, the AF protocol is preferred when both links to the destination are strong, and cooperative jamming is optimal in the high transmit power regime. The remainder of this work is organized as follows. The mathematical model of the relaying protocols is introduced in Section II. The exact and asymptotic secrecy outage proba- bilities of direct transmission, AF relaying, and cooperative jamming are derived in Sections III and IV, respectively. Selected numerical results are shown in Section V, and we conclude in Section VI. II. MATHEMATICAL MODEL We consider a half-duplex two-hop relaying system com- posed of a source (Alice), a destination (Bob), and an unau- thenticated AF relay. The channel is assumed to be quasi-static (constant during the two hops) with Rayleigh fading. We also assume all nodes in the network have the same power budget P . A. Relay Protocol We now provide the signal model for the AF relaying channel. During the first phase, the relay and Bob receive y R = α AR x A + n R (1) y B1 = α AB x A + n B1 (2) respectively, where x A is the signal transmitted at Alice with variance E{x H A x A }≤ P , α ij ∼ CN (0, ¯ γ ij ) is the complex circularly symmetric Gaussian channel gain between node i and j , with i, j ∈{A, R, B} denoting which of the three terminals is involved, and n i ∼ CN (0,N 0 ) is additive white Gaussian noise at node i. For simplicity, we assume that the