IEEE SIGNAL PROCESSING LETTERS, VOL. 17, NO. 11, NOVEMBER 2010 953 A Two-Stage Antenna Subset Selection Scheme for Amplify-and-Forward MIMO Relay Systems Heesun Park, Student Member, IEEE, and Joohwan Chun, Senior Member, IEEE Abstract—A two-stage antenna subset selection scheme is pro- posed to maximize the channel capacity of amplify-and-forward multiple-input multiple-output relay systems. To reduce the com- putational complexity of the antenna selection process, the receive and transmit antenna selections are separated in a two-stage al- gorithm. In stage 1, a subset of receive antennas is selected at the relay node. A subset of transmit antennas is then selected at the relay node in stage 2. The simulation results demonstrate that the proposed scheme achieves nearly the same channel capacity with a reduced amount of computational complexity as compared to an exhaustive search. Index Terms—Amplify-and-forward multiple-input mul- tiple-output relay, antenna selection, channel capacity. I. INTRODUCTION T HE use of multiple antennas in wireless relay systems has been considered in previous literature [1]–[3]. More specifically, amplify-and-forward (AF) multiple-input multiple- output (MIMO) relay systems (Fig. 1) have been extensively studied because AF relay can be easily implemented compared to decode-and-forward relay [2], [3], [6]–[14]. However, the cost of multiple RF chains corresponding to multiple antennas limits the deployment of AF MIMO relay systems, as in the case of MIMO systems [4], [5]. Furthermore, relays are considered a means of reducing infrastructure deployment costs [6], and thus, the number of RF chains available at the relay node is quite lim- ited in practical communication systems. Thus, for a low-cost relay system that contains a limited number of RF chains, the antenna subset selection is an attractive solution with respect to performance and complexity. Optimal antenna selection, which exhaustively checks for all possible candidates of antenna sets, exhibits an exponential in- crease in computational complexity. Therefore, an exhaustive search can hardly be considered in practice. To avoid the com- putational expense of an exhaustive search, several antenna se- lection schemes have been studied in AF MIMO relay systems [7]–[12]. In [7], [8], a transmit antenna selection at the source node (and also at the relay node in [7]) was introduced only if a single spatial stream is transmitted from the source to the destination. For a bidirectional relay system with multiple-an- tenna relay and single-antenna sources, a low complexity relay Manuscript received July 04, 2010; revised August 27, 2010; accepted September 05, 2010. Date of publication September 27, 2010; date of current version October 04, 2010. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Yong Liang Guan. The authors are with the Scientific Computing Laboratory, Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon 305-701, Korea (e-mail: hspark@sclab.kaist.ac.kr; chun@ee.kaist.ac. kr). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/LSP.2010.2079328 Fig. 1. AF MIMO relay system. antenna selection, which achieves full diversity order, was in- troduced in [9]. A relay antenna subset selection scheme was proposed to maximize a lower bound of the capacity (not the exact capacity) in [10]. However, it was assumed that the re- ceive and transmit antenna sets are the same, and thus, the per- formance is limited with respect to an optimal selection, which has no equality constraint on the receive and transmit antenna sets. If an AF relay system contains multiple relay nodes, each of which has multiple antennas and a single RF chain, the au- thors in [11], [12] suggested a greedy relay antenna selection algorithm (in which the number of selected antennas (relays) is not fixed) to minimize the mean squared error and to maximize the capacity, respectively. Furthermore, [10]–[12] exploited a naive AF scheme that simply amplifies the received signal under a power constraint at the relay node. Therefore, it is necessary to consider a relay antenna subset selection if an optimal struc- ture of the beamforming matrix is applied at the relay node [13], [14]. In this letter, we propose a two-stage relay antenna subset selection scheme to maximize the channel capacity of AF MIMO relay systems, which have an optimal beamforming structure at the relay node [13]. We separate the receive and transmit antenna selections in a two-stage algorithm by effi- ciently dividing the channel capacity into two terms. In stage 1, the receive antennas are selected to maximize the capacity of the source-relay channel. For the given receive antennas, we then select the transmit antennas that minimize the capacity loss caused by the relay-destination channel in stage 2. We also demonstrate the performance of the proposed scheme; a nearly optimal channel capacity is achieved with a very low level of computational complexity as compared to an exhaustive search. Notation: For a matrix , , , and denote the conjugate transpose, determinant, and th element of , re- spectively. The terms and represent the -di- mensional space with complex- and real-valued elements, re- spectively. For a vector , denotes the 2-norm of . The term denotes the identity matrix, and denotes the statistical expectation. II. SYSTEM MODEL We consider an AF MIMO relay system in which the source node, , has antennas; the relay node, , has antennas; and the destination node, , has antennas. The direct channel from to is assumed to be absent, due to large path loss between and . We also assume that operates in a half-duplex signaling mode. In addition, 1070-9908/$26.00 © 2010 IEEE