2758 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 62, NO. 8, AUGUST 2014 An Optimal Full-Duplex AF Relay for Joint Analog and Digital Domain Self-Interference Cancellation Young Yun Kang, Byung-Jae Kwak, and Joon Ho Cho, Member, IEEE Abstract—In this paper, a full-duplex (FD) amplify-and- forward (AF) relay is designed to compensate for the duplexing loss of the half-duplex (HD) AF relay. In particular, when there is no direct link between a source and a destination, joint analog domain self-interference suppression and digital domain residual self-interference cancellation is considered with an FD-AF relay having single receive antenna but multiple transmit antennas. Un- like previous approaches, a nonconvex quadratically constrained quadratic programming problem is formulated to find the optimal solution. The end-to-end spectral efficiency or, equivalently, the end-to-end signal-to-interference-plus-noise ratio from the source to the destination is chosen as the objective function to be max- imized subject to the average transmit power constraint at the relay. In addition, an average power constraint is imposed on the output of the relay’s receive antenna to avoid the nonlinear distortion in the low noise amplifier and the excessive quantization noise in the analog-to-digital converter. Through the systematic reduction and the partitioning of the constraint set, the optimal solution is derived in a closed algorithmic expression and shows how it allocates the transmission power not only in the direction of maximal performance improvement but also in the orthogonal direction in order to balance the system performance and the amount of self interference. It is shown that the optimal FD-AF relay significantly outperforms the optimal HD-AF relay even with the hardware limitations in the RF chain of the relay’s receiver being well taken into account. Index Terms—Full-duplex relay, amplify-and-forward relay, self-interference cancellation, nonconvex quadratically con- strained quadratic programming. I. I NTRODUCTION M ULTIHOP wireless relay has been attracting a lot of attention for its potential to extend the coverage and to improve the link quality of communication systems. Relay technologies are included in the current standard for wireless metropolitan area network such as IEEE 802.16j and IEEE Manuscript received February 16, 2014; revised June 6, 2014; accepted July 15, 2014. Date of publication July 23, 2014; date of current version August 20, 2014. This work was supported in part by the National Research Foundation funded by the Ministry of Science, ICT and Future Planning, Korea, under Basic Science Research Program Grant 2009-0088483 and under Grant 2011-0010906, and in part by the Electronics and Telecommunications Research Institute (ETRI) under Grant 2009-F-034-01. This paper was pre- sented in part at the 24th IEEE International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC), London, U.K. September 8–11, 2013. The associate editor coordinating the review of this paper and approving it for publication was A. Nallanathan. Y. Y. Kang and J. H. Cho are with the Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 790-784, Korea (e-mail: jjanguni@postech.ac.kr; jcho@postech.ac.kr). B.-J. Kwak is with the Electronics and Telecommunications Research Insti- tute (ETRI), Daejeon 305-700, Korea (e-mail: bjkwak@etri.re.kr). Digital Object Identifier 10.1109/TCOMM.2014.2342230 802.16m, and adopted by the cellular communication system such as the Third Generation Partnership Project (3GPP) Long Term Evolution (LTE)-Advanced [1], [2]. For uni-directional communication from a source to a des- tination, most of the attention has been paid so far to a half- duplex (HD) relay. This is because a full-duplex (FD) relay that transmits and receives in the same frequency band at the same time may generate a strong self interference to the relay itself [3], [4]. Such a strong self interference may saturate the low noise amplifier (LNA) in the relay’s receive radio frequency (RF) chain. In addition, the huge difference in signal level by several tens of dB or more between the desired signal and the self interference at the relay’s receive antenna requires an increased number of bits of resolution from the analog-to- digital converter (ADC) that follows the LNA. Similar phe- nomenon occurs when any full duplex system is considered [5]. Instead, the HD relay utilizes two orthogonal channels, e.g., in the time or in the frequency domain in order to transmit and receive without any effect of self interference. However, such orthogonal channel allocation results in a significant duplexing loss in spectral efficiency. For bi-directional communication, a two-way relay [6] as well as the HD relay can be employed to overcome the self- interference problem. The two-way relay also operates in the HD mode, by using the two orthogonal channels now for multiple-access and broadcasting to and from the relay, re- spectively. Thus, the two-way relay has a disadvantage in that its deployment requires change in frame structures from the system without a relay, as that of the HD relay does. Moreover, it introduces multi-user interference that must be suppressed to properly compensate for the duplexing loss. Recently, with the advent of multi-antenna technologies, there has been a regained interest in the use of the FD relay that does not require two orthogonal channels [7]–[10]. In [7], the maximization of the end-to-end information rate from the source to the destination is considered for the FD amplify-and- forward (AF) relay with multiple transmit and receive antennas. The optimal transformation matrix of the relay is derived under the assumption that the self interference is perfectly cancelled by time-domain digital signal processing. In [8], the minimiza- tion of the self interference is considered for the FD-AF relay with multiple transmit and receive antennas. The jointly optimal transmit and receive filters are derived under the assumption that all means have been already taken to improve natural isolation between the transmit and receive antennas. However, possible nonlinear distortion in the LNA and finite precision effect in the ADC due to the strong self interference are ignored in [7] and [8]. 0090-6778 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.